For those hoping to discover groundbreaking new products, you might be left wanting. The real buzz in recent years seems to center around Open AI’s announcements, with their innovations having the potential to disrupt entire industries. The multitude of products on display here leverage less glamorous technology in new combinations to address existing or potential problems more efficiently, more powerfully, lighter, and sleeker. If you have something new that can help consumers, this is the place to showcase it.
I’m not an avid electronics enthusiast, but rather an early majority pragmatist on the Technology Adopt Lifecycle who cares about solutions that genuinely solve problems. I casually observed categories like cars, TVs/monitors, speakers and headphones, fitness bands, and medical devices, enjoying the fair-like atmosphere—your car can fly and dig, your robotic arm can make latte art, but what does that have to do with me? However, appreciating the design of electronic products, the sheer excitement and fun of sound and light, and understanding what electronic machinery can achieve at certain costs and price points is enlightening.
I found the startup and AI areas most intriguing and paid them the most attention. Software, FinTech, Web & Mobile Apps, and productivity tools for personal and corporate use were interspersed among the various hardware solutions.
Rabbit R1 appears to be the only notable portable LLM hardware, lacking a booth but generating significant interest. It’s puzzling why one would need a dedicated hardware for AI apps when a smartphone can perform the same functions, and possibly even better. Insights from experts are welcome.
booth.ai employs generative AI to assist e-commerce in photo editing: upload product photos and input text descriptions to generate images, such as adding backgrounds, changing people, or altering poses.
AiD / Cream creates detailed comic images and colors based on artists’ sketches. A demo I tried showed impressive results, although the comics weren’t overly complex.
Plaud Note, a card attached to the back of a phone for real-time recording, addresses the iPhone’s inability to record and uses AI to quickly generate summaries. The product is well-made but not cheap.
vcat.ai generates marketing videos from the details of your product webpage. Although it sounds fancy, the URL merely imports materials from the web page, requiring manual template selection and adjustments in their system afterward.
Keeneat.com offers background music generation for creative professionals, along with a marketplace. This seems marketable, given the increasing number of video creators and the limited, expensive stock music available.
Blovo ChatGPT for animals allows you to inquire about anything pet-related.
FinTech’s visibility at CES was surprisingly low, with only five companies present, one of which was absent. The largest booth belonged to South Korea’s Shinhan Bank, proud of their many years at CES. Their interest in CES stems from showcasing cashier machines as human teller replacements, aligning with the trend towards banking digitization and automation. The biggest challenge mentioned was adapting the UI/UX for older users, as high-net-worth VIP customers are not their primary target.
Woongjin is Korea’s leading rental and subscription management system, integrating finance, payment, and insurance solutions, serving many large Korean conglomerates.
veintree (press release) offers identity verification through vein patterns on hands, claiming advantages over iris recognition in terms of anonymity and not requiring specialized equipment. It’s being explored as a factor in multi-factor authentication, similar to Windows Hello and Face ID, and is seeking blockchain partnerships, like with Solana, to create a hand-based Worldcoin.
Fog Hashing sells Bitcoin mining machines for home use and professional data center solutions. Due to high electricity costs in California, they recommend mining in other states.
Flux token, an L1 PoW blockchain, supports distributed data centers running data storage, CMS, docker droplets, etc., serving as a platform for censorship-resistant DApps.
Pay.cool is an open-source decentralized crypto payment network from China, enabling consumers to pay merchants with cryptocurrencies. The app’s design and their unclear offerings were noted during a conversation, though they are seeking mining partners.
DeluPay from France offers crypto payment solutions. Only their business card was seen, with their website in French, indicating incomplete internationalization.
There were numerous Sleep Tech products, such as motionsleep’s snore-control pillows, bokuk’s temperature-regulating duvets, frenz’s brainwave-monitoring and bone-conduction music headbands, and Lumos tech’s sleep masks.
eclypia introduced a non-invasive glucose-monitoring watch.
aqara uses AI and spatial technology to improve elderly care.
Maintaining good physical health and sleep habits is an effortless way to save money. Good health is invaluable.
Exhibit location matters: Avoid corners, small meeting rooms, and dead-ends with low foot traffic. Being in the main hall with popular exhibitors is more effective.
Pre-scheduling meetings and using hotel suites for product displays or business discussions can be a cost-effective strategy.
Hardware is easier to exhibit, especially with performance aspects, unlike software, which relies on engaging taglines, flyers, and demos to attract attention. Descriptive scenes and explanations are essential for less obvious hardware.
Exhibitors engrossed in their phones or in private conversations deter visitor engagement.
Sharing a booth among similar businesses, though unofficial, can help manage budgets.
Booking CES space early is crucial; arrangements can usually be made if plans change.
Branding should be visible to assist visitors with note-taking and photos.
If your demo is impressive, include your booth location on materials to attract more visitors through secondary sharing.
Keep an open mind: Exhibitors are seeking industry connections. Criticism should be reserved as many are there to find or offer solutions within the industry.
The Innovation Awards can help narrow down interests amid the overwhelming variety of products.
Evening events and after-parties offer leisure and networking opportunities distinct from the more formal daytime exhibitions.
Protect your CES badge; replacements are costly.
South Korea’s presence was notably strong, driven by their need for international markets. Their broad category representation and the dedication of individual exhibitors, including non-English speakers, were impressive.
Apple’s absence makes sense, given their ecosystem could overshadow many smaller exhibits.
SQPV glass presents solar panels as glass, though the appearance is somewhat grey, potentially fitting for skyscrapers with lower light transmission requirements.
ProtoHologram impressively projects holograms into a display box.
Elon Musk’s Boring Company has a loop under LVCC, offering a unique but ordinary tunnel ride experience, only made notable upon realization of its significance.
For consumers, CES 2024 is a haven for electronic enthusiasts to experience and purchase products at discounted rates. It may not cater well to those already familiar with the market and looking for niche advancements.
For businesses, it’s an ideal venue for hardware manufacturers and e-commerce to build supply chains and find collaborations, less so for pure software companies.
]]>A Diminisher, in contrast to an Illuminator, is a person who negatively impacts those around them through their interactions and behaviors. They tend to make others feel small, unimportant, or unseen, often focusing self-centeredly on their own needs and interests.
To be an effective conversationalist or an “illuminator,” one must focus on deeply engaging and understanding others. Here are key takeaways:
Being a good conversationalist is about creating a mutual journey of exploration, understanding, and deep listening, where both parties feel heard and valued.
]]>Doordash:
Rippling Employee Onboarding:
Shift from “blockchain-powered” to “AI-powered” company claims.
Emphasis on AI, machine learning, and blockchain as tools, not business models. Their value depends on their application in products and economic models.
Generative AI’s potential in reducing labor costs and the challenge to differentiate genuine AI usage from PR hype.
2. Industry Insights:
Insights from Cannabis Banking Summit: Challenges and opportunities in cannabis banking. Engagement with cannabis leading banks and credit unions to enhance risk management and compliance.
Pay-by-bank has emerged as a popular payment method that offers a convenient and secure way for customers to make online purchases directly from their bank accounts.
3. Political and Regulatory Influence:
4. Banking as a Service (BaaS) and Open Banking:
5. Compliance, Compliance, Compliance. Dodd-Frank Act Section 1033 – Consumer Access to Financial Records
Translator: BlockEden.xyz Team and Payton Chat
📌 A deep dive into the regulatory disputes and legal issues the crypto industry faces in the past, now, and predictably in the future.
TL;DR
On July 13, 2023, Ripple Labs received a partial favorable ruling from the New York District Court, causing a significant surge in the crypto market. In addition to XRP itself, a series of tokens previously identified as securities by the SEC also experienced a substantial increase.
As we will discuss later, we are still far from the era when the crypto industry truly embraces clear regulation. However, without a doubt, this partial victory of Ripple Labs remains one of the most important events in the crypto industry in 2023.
Below are some of the major disputes between U.S. regulators and the crypto industry before the SEC vs. Ripple Labs case.
Case | Date Settled | How it’s Settled |
---|---|---|
SEC vs Block.one (EOS) | 2019/09 | Block.one Settles with SEC, Pays $24mn Fine |
SEC vs Telegram | 2020/06 | Court Rules Telegram’s Actions as Selling Unregistered Securities, Telegram Returns $1.2bn to Investors and Pays $18.5mn Fine |
CFTC vs BitMEX | 2021/08 | Court Determines BitMEX Engaged in Illegal Derivative Trading (specific projects are too numerous to elaborate), BitMEX Pays $100mn Fine and Ceases Illegal Activities |
SEC vs BlockFi | 2022/02 | BlockFi Settles with SEC, Seeks Business Compliance, and Pays $100mn Fine |
SEC vs Nexo | 2023/01 | Nexo Settles with SEC, Shuts Down Lending Business, and Pays $45mn Fine |
SEC vs Kraken | 2023/02 | Kraken Settles with SEC, Shuts Down Staking Business, and Pays $30mn Fine |
CFTC vs Ooki DAO | 2023/06 | Court Determines Ooki DAO as an Illegal Futures Trading Platform, Orders to Shut Down All Business, and Pays a $644k Fine |
It’s not hard to see that nearly all the major disputes so far have ended in failure or compromise by crypto companies.
We still want to say, this represents the first meaningful victory for the crypto industry in its battles against U.S. regulators, even if it is only a partial victory.
There have been many detailed interpretations of the court’s judgment, so we won’t elaborate here. Those who are interested can read the long Twitter thread by Justin Slaughter, Paradigm Policy Director:
Ok, having gone through the Ripple decision, here’s my takeaway:
— Justin Slaughter (@JBSDC){" "} July 13, 2023
Big loss for the SEC’s approach to crypto via focusing solely on enforcement, and this measurably increases the odds of crypto legislation passing this year.
Thread https://t.co/c4wOVPORVb
You can also read the original text of the court’s ruling in your leisure time:
Plaintiff vs. Ripple Labs, Inc.
Before further interpreting this ruling, let’s briefly introduce the core standard for the definition of securities in the U.S. legal system that you often hear about, the Howey Test.
To understand the disputes surrounding all cryptocurrency regulations today, we must go back to sunny Florida in 1946, to the cornerstone case for today’s securities law judgment, SEC vs. Howey.
(The following story outline was mainly written with the help of GPT-4)
📌 After World War II, in 1946, the company W.J. Howey owned a fertile orange grove in picturesque Florida.
To raise more investment, the Howey company launched an innovative plan that allowed investors to purchase land in the orange grove and lease it to the Howey company for management, from which investors could earn a portion of the profits. In that era, this proposition was undoubtedly very attractive to investors. After all, owning your own land was such a tempting thing.
However, the SEC did not agree. The SEC believed that the plan offered by Howey Company was essentially a security, but Howey Company had not registered with the SEC, which clearly violated the Securities Act of 1933. Therefore, the SEC decided to sue the Howey Company.
This lawsuit eventually ended up in the Supreme Court. In 1946, the Supreme Court made a historic judgment in the lawsuit of SEC vs. Howey. The court supported the SEC’s stance, ruling that Howey Company’s investment plan met the definition of securities, and therefore needed to be registered with the SEC.
The U.S. Supreme Court’s judgment on Howey Company’s investment plan was based on the four basic elements of the so-called “Howey Test”. These four elements are: investment of money, expectation of profits, common enterprise, and the profits come from the efforts of the promoter or a third party. Howey Company’s investment plan met these four elements, so the Supreme Court determined it was a security.
First, investors invested money to purchase land in the orange grove, which met the first element of the “Howey Test”—investment of money.
Secondly, the purpose of investors buying land and leasing it to the Howey Company was obviously to expect profits, which met the second element of the “Howey Test”—expectation of profits.
Third, the relationship between investors and the Howey Company constituted a common enterprise. Investors invested, and the Howey Company operated the orange grove, both working towards earning profits. This met the third element of the “Howey Test”—common enterprise.
Lastly, the profits in this investment plan mainly came from the efforts of the Howey Company. Investors only needed to invest money and could reap the benefits, which met the fourth element of the “Howey Test”—the profits come from the efforts of the promoter or a third party.
Therefore, according to these four elements, the Supreme Court judged that Howey Company’s investment plan constituted a security and needed to be registered with the SEC.
This judgment had profound implications and formed the widely cited “Howey Test”, defining the four basic elements of so-called “investment contracts”: investment of money, expectation of profits, common enterprise, and profits come from the efforts of the promoter or a third party. These four elements are still used by the SEC to determine whether a financial product constitutes a security.
For purposes of the Securities Act, an investment contract (undefined by the Act) means a contract, transaction, or scheme whereby a person invests his money in a common enterprise and is led to expect profits solely from the efforts of the promoter or a third party, it being immaterial whether the shares in the enterprise are evidenced by formal certificates or by nominal interests in the physical assets employed in the enterprise.
The above is an accurate interpretation of securities from the 1946 Supreme Court opinion, which can be broken down into the following commonly used criteria:
The charm of law is truly remarkable. It often employs abstract yet straightforward principles to guide the ever-changing specificities in real-life scenarios, no matter it is a citrus grove or cryptocurrency.
In fact, how securities are defined is not important. Labeling something as a security or not doesn’t make any substantive difference. The key is to understand what legal responsibilities stem from the economic nature of securities, in other words, why something possessing the four attributes of the Howey Test needs a separate legal framework for supervision.
The Securities Act of 1933, which predates the Howey Test by over a decade, explicitly answers the question of why securities laws are needed.
Often referred to as the “truth in securities” law, the Securities Act of 1933 has two basic objectives:
1) require that investors receive financial and other significant information concerning securities being offered for public sale; and
2) prohibit deceit, misrepresentations, and other fraud in the sale of securities.
"The fundamental starting point of securities law is simple - it’s all about ensuring that investors have enough information about the securities they are investing in and are protected from deception. Conversely, the responsibilities imposed on the issuers of securities are straightforward, the essence of which is disclosure - they must provide complete, timely, and accurate disclosure of important information related to the securities.
The reason for such a goal of securities law is because securities, by their nature, rely on the efforts of third parties (active participants) for returns, which gives these third parties an asymmetric advantage over investors in terms of access to information and influence on securities prices. Therefore, there’s a requirement for them to fulfill the duty of disclosure, to ensure that this asymmetry does not harm the investors.
There’s no similar regulatory requirement in commodities markets because there are no such third parties, or in the crypto context, ‘project teams’. Gold, oil, and sugar, for example, have no ‘project teams’. The crypto market generally has a preference for the Commodity Futures Trading Commission (CFTC) over the Securities and Exchange Commission (SEC), but this is not due to personal preferences of the regulators that lead to differing attitudes towards crypto. The distinction between regulating commodities and regulating securities is based on the intrinsic differences between the two types of financial products. Because there are no ‘project teams’ with an asymmetric advantage, the regulatory framework for commodity law naturally tends to be more relaxed.
💡 The existence of a third party or ‘Project Team’ with an information and influence advantage is the fundamental reason for the existence of securities law; to curb the infringement of investors’ interests by the third party/‘Project Team’ is the fundamental purpose of securities law; and requiring the ‘Project Team’ to provide complete, timely, accurate information disclosure is the main means of implementing securities law."
During my study of the history of U.S. securities law, a phrase often heard in the crypto industry led me to a simple and effective standard to determine whether a token is a security - that is, whether the investor cares whether the Project Team is active or not.
If the “the project team is doing their job” matters to investors, it implies that the return on this investment is influenced by the actions of the Project Team, which clearly meets the four criteria of the Howey Test. From this perspective, it’s easy to understand why BTC is not a security, as there is no Project Team involved with BTC. The same applies to meme coins, they are merely digits in the ledger under the ERC-20 protocol, with no active Project Team behind them, and therefore are not securities.
If a Project Team is active and whether they perform well or poorly, or act at all, - whether it’s in terms of technical upgrades, product iterations, marketing, ecosystem partnerships - has an impact on the token price, then the definition of a security is met. Given the existence of a Project Team, they possess information unknown to other investors and have greater influence on the token price, hence the need for regulatory oversight to ensure that they do not commit acts that harm the interests of investors. The logic of “the actions of the Project Team matter” → "the Project Team can reap the benefits"→ “the Project Team needs to be regulated by securities law” is a simple legal inference.
If you accept this logic, you can judge for yourself which tokens in the crypto space are reasonably classified as securities.
top search result of “项目方在做事” on Twitter
💡 In our view, if there is an expectation or concern among investors about the “the project team is doing their job,” this token highly aligns with the definition of a security. From this perspective, it seems quite logical that a high proportion of tokens are classified as securities.
The current SEC wants more than just the basic regulations. As seen from Gary’s public statements, he only recognizes that Bitcoin is not a security. For most other tokens, he firmly believes they should be classified as securities. The stance on a few tokens, like ETH, is relatively ambiguous. The CEO of Coinbase also recently mentioned in an interview that before the SEC sued Coinbase, it had demanded that Coinbase cease trading all tokens except for Bitcoin, a request that Coinbase refused.
We think it’s unreasonable to classify pure meme coins without an operational project team or decentralized payment tokens as securities. The SEC’s demands have exceeded the reasonable scope of securities laws, which has made it harder for the conflict between the industry and the SEC to be resolved simply.
You can read more on the topic in this article: SEC asked Coinbase to halt trading in everything except bitcoin, CEO says."
Howey Test’s Rules | Analysis |
---|---|
1. An investment of money |
✅ It satisfies the criteria; institutional investors made payments to XRP, and Ripple Labs argued that not only is ‘payment of money’ required, but also ‘an intent to invest’. This claim was rejected by the court. |
2. in a common enterprise |
✅ It satisfies the criteria; the funds invested by the investors were collectively received and managed by Ripple Labs, and what the investors received were the same fungible XRP tokens. |
3. to expect profits |
✅ It satisfies the criteria; 1) All the promotional materials from Ripple received by the investors clearly mention in various ways that the success of the Ripple protocol would drive up the price of XRP. 2) The existence of the lock-up clause directly proves that the investors’ intent in purchasing XRP could only be investment and not consumption (‘a rational economic actor would not agree to freeze millions of dollars’). |
4. solely from the efforts of the promoter or a third party |
✅ It satisfies the criteria; Ripple Labs explicitly linked the rise in XRP price to the technical advantages of Ripple Labs, the potential for widespread use of the product, the professional capabilities of the team, and successful market marketing in its promotions. |
The reasons for judging programmatic sales as not constituting securities sales are:
In this case, investors are not sure whether they are buying from Ripple Labs or other XRP sellers. Most XRP trading volume does not come from sales by Ripple Labs, so most XRP buyers have not directly invested their funds into Ripple Labs.
XRP buyers did not expect to profit from Ripple Labs’ efforts, because:
Ripple Labs did not make any direct promises to these investors, and there is no evidence that Ripple Labs’ promotional materials were widely disseminated among these investors.
These investors are less sophisticated, and it cannot be proven that they have a full understanding of the impact of Ripple Labs’ actions on the price of XRP.
It’s not hard to see that the court’s judgement on programmatic sales is primarily based on the fourth item of the Howey Test, which is that these investors did not expect to profit from Ripple Labs’ efforts.
The judgement of this district court does not have final binding force; it can almost be certain that the SEC will appeal. However, due to the lengthy legal process, it might take several months or even years before we see the results of a new appeal judgement. During this time, the judgement of this court will essentially form important guidance for the development of the industry.
Putting aside our position as cryptocurrency investors, and solely from the standpoint of legal logic, we believe that the court’s logic in determining programmatic sales as not being securities is not very convincing.
📕 Here are two articles by seasoned legal professionals with similar opposing views. I recommend reading them if you have time, as our analysis also draws on some of their viewpoints.
First, we need to note the original text of the Howey Test: ‘…expect profits solely from the efforts of the promoter or a third party…’, which clearly points out that the source of profits can be the promoter or a third party, that is, it does not matter who the seller is. Or to say, it is not necessary for the source of the efforts to be the seller or promoter, as long as there is such a third party. Therefore, it does not matter who the investor buys from or whether the seller is the source of the returns. What matters is whether the investor realizes that the appreciation of the asset comes from the efforts of a third party. Therefore, the court’s mention of blind buy/sell and the fact that buyers do not know whether they bought XRP from Ripple Labs or someone else is irrelevant to the Howey Test.
The real issue is whether investors in programmatic sales realize that the rise in the price of the XRP token they bought is related to the efforts of Ripple Labs. The court’s main argument is that
First of all, this is a factual issue, not a logical one, which we can’t demonstrate here. XRP is an old project, and we don’t have a clear sense of what the retail investors were like at that time.
But from our limited experience, the vast majority of tokens with a project team are able to realize that the team’s technical upgrades, early mainnet launch, better product, increase in TVL, ecosystem partnerships, KOL promotions, and other efforts have an impact on the price of the token they hold.
In the world of crypto, KOLs, Twitter, and Telegram groups large and small serve as the bridge between most project teams and users, the territory for outreach to retail investors. In projects big and small, we often hear discussions about how the ‘community’ is doing. Most project teams will have a token marketing/community team responsible for contacting exchanges around the world, hiring KOLs, and helping to disseminate project progress and important events.
💡We believe there is a bias in the court’s fact-finding on programmatic sales in this ruling; we also agree with many legal professionals that there is a high likelihood that this part of the judgment will be overturned in the future.
(Just a week after writing this article, on the very day it was about to be published, we happened to see that the new judge in the SEC vs Terraform Labs case refused to adopt the judgment logic in the SEC vs Ripple Labs case - the logic being that no matter where the investor buys the token, it does not affect the investor’s expectation that the efforts of the project team will influence the token’s price.)
“Whatever expectation of profit they had could not, according to that court, be ascribed to defendants’ efforts,” he wrote. "But Howey makes no such distinction between purchasers*. And it makes good sense that it did not. That a purchaser bought the coins directly from the defendants or, instead, in a secondary resale transaction* has no impact on whether a reasonable individual would objectively view the defendants’ actions and statements as evincing a promise of profits based on their efforts.**"
— Judge Rejects Ripple Ruling Precedent in Denying Terraform Labs’ Motion to Dismiss SEC Lawsuit
☕️ By the way - Airdrops that don’t require payment can also be considered securities sales.
This comes from an article by John Reed Stark. In the Internet bubble of the late 90s, several companies distributed free stocks to users via the internet. In subsequent legislation and trials, these actions were deemed securities sales. The reason is that although users did not pay money in exchange for these stocks, they gave up other values - including their personal information (required to fill in when registering for stocks) and increased attention for the companies distributing the stocks, which constituted a substantial exchange of value.
SEC Enforcement Director Richard H. Walker said at the time, "Free stock is really a misnomer in these cases. While cash did not change hands, the companies that issued the stock received valuable benefits*. Under these circumstances, the securities laws entitle investors to full and fair disclosure, which they did not receive in these cases.”*
Don’t be misled that Judge Torres ruled that sometimes XRP is a security and sometimes it isn’t. That’s exactly the opposite of what she ruled: XRP itself is NEVER a security. “ Page 15: "XRP, as a digital token, is not in and of itself a ‘contract, transaction[,] or scheme’…
— paulgrewal.eth (@iampaulgrewal){" "} July 14, 2023
As pointed out by Coinbase CLO Paul, this is the most important sentence in the entire judgement that people have not fully understood.
XRP, as a digital token, is not in and of itself a “contract, transaction[,] or scheme” that embodies the Howey requirements of an investment contract*. Rather, the Court examines the* totality of circumstances surrounding Defendants’ different transactions and schemes involving the sale and distribution of XRP.
Both of these judgments consistently express an important point of view:
A token is just a token - it’s not like many people mistakenly believe that the court sometimes thinks XRP is a security and sometimes not - a token itself can never be a security.
What might constitute a security is the whole set of behaviors of selling and distributing tokens (‘scheme’), there is no question of whether a token is a security or not, only whether a specific token sale behavior is a security or not. We can never come to the conclusion of whether it is a security or not just by analyzing a certain token, we must analyze the overall situation of this sales behavior (‘entirety of …’, ‘totality of circumstances’).
Both judges, whose opinions have significant conflicts, have insisted that it must be based on sales conditions rather than the attributes of the token itself to determine whether it is a security - this consistency also means that the possibility of this legal logic being adopted in the future is significantly higher than the judgment for programmatic sales, and we also believe that this judgment indeed has stronger logical reasonableness.
A token is just a token. A token is NEVER a security.
Digital tokens and stocks are fundamentally different. Stocks themselves are a contract signed by investors and companies. Their trading in the secondary market itself represents the trading and transfer of this contractual relationship. As the judge said in the Telegram case, digital tokens are nothing more than an ‘alphanumeric cryptographic sequence’, and they cannot possibly constitute a contract by themselves. They can only have the economic substance of a contract in specific sales situations.
If this legal point of view is accepted by all subsequent courts, then the future burden of proof on the SEC in the litigation process will be significantly increased. The SEC cannot obtain the regulatory power over all the issuance, trading, and other behaviors of a certain token by proving that it is a security. It needs to prove one by one that the overall situation of each token transaction constitutes a securities transaction.
The Court does not address whether secondary market sales of XRP constitute offers and sales of investment contracts because that question is not properly before the Court. Whether a secondary market sale constitutes an offer or sale of an investment contract would depend on the totality of circumstances and the economic reality of that specific contract, transaction, or scheme. See Marine Bank, 455 U.S. at 560 n.11; Telegram, 448 F. Supp. 3d at 379; see also ECF No. 105 at 34:14-16, LBRY, No. 21 Civ. 260 (D.N.H. Jan. 30, 2023)*
The Ripple case also explicitly pointed out that the court cannot determine whether the secondary sale of XRP constitutes a securities transaction. They need to assess the specific situation of each trading behavior to make a judgment. This greatly complicates the SEC’s regulation of secondary transactions, and in some ways it may not be possible to complete; this essentially gives the green light to the secondary trading of tokens. Based on this, Coinbase and Binance.US quickly relisted XRP after the verdict was announced.
📕 There are some interesting discussions related to this in the Bankless podcast:
Bankless: How Ripple’s Win Reshapes Crypto with Paul Grewal & Mike Selig
Again, it is still too early to consider this judgment as a definitive legal rule based solely on this case; but the legal logic of “A token is just a token” will indeed significantly increase the legal obstacles the SEC will face in regulating transactions of the secondary market in the future.
Sword of Damocles, 1812, Richard Westall
ETH staking has been one of the strongest tracks in the entire industry since 2023; however, the regulatory risks of staking services are still a Sword of Damocles over this super track.
In February 2023, Kraken agreed to a settlement with the SEC and shut down its staking service in the US. Coinbase, which was also sued for its staking service, chose to continue fighting.
Returning to the framework of the Howey Test, objectively speaking, there are indeed sufficient reasons for staking services to be considered securities.
Howey Test’s Rules | Analysis |
---|---|
1. An investment of money |
✅ It satisfies the criteria; invest ETH |
2. in a common enterprise |
✅ It satisfies the criteria; invested ETHs are pooled together |
3. to expect profits |
✅ It satisfies the criteria; Investors expects staking yields |
4. solely from the efforts of the promoter or a third party |
✅ It satisfies the criteria; staking yields come from the node operator’s work and the node operator charges commission from the work. |
Kraken chose to settle. So, what are Coinbase’s reasons for insisting that staking services are not securities?
Coinbase: Why we stand by staking:
At its most basic level, staking is the process by which users can contribute to the network by staking their token to secure the blockchain, facilitate the creation of blocks, and help process transactions. Users are not investing. Rather, users are compensated for fulfilling this important role through transaction fees and consensus rewards paid by the blockchain itself.
Coinbase makes an interesting statement, suggesting that “users who stake are not investing, but rather being compensated for the contribution they make to the blockchain network.”
This statement is valid for individual stakers. However, as delegated stakers, they do not directly undertake the task of validating transactions or ensuring network security. Instead, they delegate their tokens to other node operators who use their tokens to complete these tasks. Stakers are not the direct laborers. In fact, they resemble the buyers of orange farm in the Howey case, owning land/capital (ETH), delegating others to cultivate (node operation), and obtaining returns.
Paying out capital is not labor, because the return from capital investment is a capital gain, not compensation.
Decentralized staking services are a bit more complex, and different types of decentralized staking might eventually receive different legal judgments.
The four criteria of the Howey Test are mostly similar in centralized staking and decentralized staking. The difference might lie in whether a common enterprise can exist. So, the staking model where all users’ ETH is put into the same pool, even if it’s decentralized, clearly also meets the four criteria of the Howey Test.
The argument in SEC vs Ripple Labs that allowed Ripple to win the Programatic Sales point (the buyer and seller don’t know each other and there is no direct selling introduction), doesn’t seem to protect staking services here neither.
Because apart from directly buying cbETH/stETH on the secondary market, in the case where stakers pledge their ETH to Coinbase/Lido and receive cbETH/stETH in return, it’s clear that 1) the buyer knows who the issuer is, and the issuer also knows who the buyer is, and 2) the issuer clearly communicates to the buyer about the potential returns and explains the source of these returns.
Similarly, in addition to staking on PoS chains, many DeFi products that allow staking/locking tokens to earn yield are likely to meet the definition of securities. If it is somewhat challenging to establish a connection between the price of pure governance tokens and the efforts of the project team, the logic in the context of staking to earn yield is very straightforward and simple. Additionally, the reasoning in the Ripple case that made programmatic sales not considered securities also hardly stands here:
1) Users hand over tokens to staking contracts developed by the project team. The staking contract gives returns to users, and these returns are derived from the revenues generated by the project contracts that the project team opened.
2) During the interaction process between users and the staking contract, the contract also promotes and explains the returns to users, which makes it difficult to get away with the reasoning from XRP’s programmatic sales.
💡 In summary, projects that offer staking services (in PoS chains, in DeFi projects) have a higher likelihood of being classified as securities due to
- clear profit distribution, and
- direct promotion and interaction with users.
This makes them more likely to be considered securities than projects that are generally “doing their job” by the project team.
Securities law is the main focus of this article, but it’s important to remind everyone that securities law is only a small part of the overall regulatory framework for crypto — of course, it’s worth special attention because it is one of the stricter aspects. Whether a token is ultimately regarded as a security, commodity, or something else, some more fundamental legal responsibilities are common, and many regulatory agencies outside of the SEC and CFTC will get involved. The content involved here is worthy of another long article, we will just briefly give an example here for reference.
This is the responsibility related to Know Your Customer (KYC) centered on anti-money laundering (AML) and counter-terrorist financing (CTF). Any financial transaction must not be used for financial crimes such as money laundering and terrorist financing, and any financial institution has the responsibility to ensure that the financial services it provides will not be used for these financial crimes. To achieve this goal, all financial institutions must take a series of measures, including but not limited to KYC, transaction monitoring, reporting suspicious activities to regulators, maintaining accurate records of historical transactions, etc.
This is one of the most fundamental, undisputed basic laws in financial regulation, and it is a field jointly supervised by multiple law enforcement departments, including the Department of Justice, Treasury/OFAC, FBI, SEC, etc. Currently, all centralized crypto institutions are also complying with this law to perform necessary KYC on all customers.
The main potential risk in the future lies in DeFi, whether it is necessary and possible to make DeFi comply with similar regulations as CeFi, requiring KYC/AML/CTF; and whether this regulatory model might harm the foundation of blockchain value, permissionlessness.
From a basic principle point of view, financial transactions are generated in DeFi, so these financial transactions need to ensure that they are not used for money laundering and other financial crimes, so the necessity of regulatory law is undoubted.
The challenge mainly lies in the difficulty in defining the regulatory object, essentially these financial transactions are based on the services provided by a string of code on Ethereum, so is it the Ethereum nodes running this code, or the project parties/developers who wrote this string of code, who should be the regulatory object? (That’s why there are controversial cases caused by the arrest of Tornado Cash developers.) In addition, the decentralization of nodes and the anonymization of developers make this oversight thinking even more difficult to implement — this is a problem that legislators and law enforcers must solve, it is questionable how they will solve these problems; but what is unquestionable is that no regulator will allow money laundering, arms trading and other activities on an anonymous blockchain, even if these transactions account for less than one ten-thousandth of the blockchain transactions.
Actually, just on the 19th of this month, four senators from the U.S. Senate (two Republicans and two Democrats, so it’s a bipartisan bill) have proposed a legislation for DeFi, the Crypto-Asset National Security Enhancement and Enforcement (CANSEE) Act. The core is to require DeFi to comply with the same legal responsibilities as CeFi:
In an effort to prevent money laundering and stop crypto-facilitated crime and sanctions violations, a leading group of U.S. Senators is introducing new, bipartisan legislation requiring decentralized finance (DeFi) services to meet the same anti-money laundering (AML) and economic sanctions compliance obligations as other financial companies*, including centralized crypto trading platforms, casinos, and even pawn shops. The legislation also modernizes key Treasury Department anti-money laundering authorities, and sets new requirements to* ensure that “crypto kiosks” don’t become a vector for laundering the proceeds of illicit activities.
— Bipartisan U.S. Senators Unveil Crypto Anti-Money Laundering Bill to Stop Illicit Transfers
Ensuring Anti-Money Laundering (AML) and Counter-Terrorist Financing (CTF) in DeFi transactions is a key regulatory challenge beyond securities laws. Regardless of whether a token is classified as a security or commodity, there are strict rules against market manipulation. Resolving these issues in crypto is a future challenge for the industry.
Below are some typical forms of market manipulation. Anyone involved in crypto trading will likely recognize them.
Here are some common forms of market manipulation:
- Pump and Dump: This involves buying a security at a low price, artificially inflating its price through false and misleading positive statements, and then selling the security at the higher price. Once the manipulator sells their shares, the price typically falls, leaving other investors at a loss.
- Spoofing: This involves placing large buy or sell orders with no intention of executing them, to create a false appearance of market interest in a particular security or commodity. The orders are then canceled before execution.
- Wash Trading: This involves an investor simultaneously buying and selling the same financial instruments to create misleading, artificial activity in the marketplace.
- Churning: This occurs when a trader places both buy and sell orders at the same price. The orders are matched, leaving the impression of high trading volumes, but no net change in ownership.
- Cornering the Market: This involves acquiring enough of a particular asset to gain control and set the price on it.
- Front Running: This occurs when a broker or other entity enters into a trade because they have foreknowledge of a big non-publicized transaction that will influence the price of the asset, thereby benefiting from the price movement.
We lack sufficient legal and political knowledge to predict the outcomes of these legal disputes, but objective analysis leads us to acknowledge that the logic of U.S. securities law supports classifying most tokens as securities. So we must deduce or imagine what the crypto industry might look like if most tokens are considered securities.
Firstly, purely from an economic perspective, the compliance cost of being publicly listed isn’t as daunting as it might seem. For large-cap tokens with a FDV of over 1 billion, they are likely able to bear the cost.
A simple market value comparison reveals that many tokens have comparable market values to listed companies, especially those with a 1bn+ FDV. It’s entirely reasonable to believe that they can handle the compliance costs of a listed company.
We can also refer to some research on the compliance cost for listed companies. One relatively reliable source is the SEC’s estimation of the listing compliance costs for small and medium-sized companies:
Their research shows that the average cost of achieving regulatory compliance to enter the marketplace as an IPO is about $2.5 million. Once they are established, small-cap companies can expect to pay about $1.5 million in ongoing compliance costs every year.
The conclusion is that there is a ~2.5mn listing cost, and a ~1.5mn ongoing annual cost. Considering inflation over the years, 3-4mn for an IPO and 2-3mn for annual recurring costs seem reasonable estimates. Additionally, these numbers positively correlate with the size of the company, and the costs for microcap companies worth hundreds of millions of dollars should be below these averages. Although it’s not a small amount, for large project teams with hundreds of members, it’s not an unacceptable cost."
"What’s more uncertain is how to resolve these projects’ historical compliance issues.
Listing a stock requires an audit of the company’s financial history. Tokens, unlike equity, would need to disclose different content for listing, thus requiring a new regulatory framework for clear delineation. However, as long as there are clear rules, there are ways to adjust and deal with them. Companies with historical financial problems can also get the chance to go public by restating their historical statements.
While the cost of compliance is acceptable, it is also quite high; so, are project parties incentivized to do so? There’s no simple answer to this question.
Firstly, compliance will indeed impose many burdens on many project parties and limit their operational flexibility. They cannot engage in “market value management,” insider trading, false advertising, and coin selling announcements, etc. These restrictions affect the fundamentals of many business models.
However, for projects with particularly large market values, gaining greater market liquidity, accessing more deep-pocketed investors, and obtaining comprehensive regulatory approval are essential conditions for them to move to the next level, whether from the perspective of market value growth or project development.
“‘Illegal harvesting’ can be fierce, but the ‘leek field’ is small; ‘legal harvesting’ must be restrained, but the ‘leek field’ is large.”
As the project scale increases, the balance between the potential benefits of non-compliance and the opportunities brought by the vast market and capital access post-compliance increasingly tips towards the latter. We believe that leading public chains/layer2s and blue-chip DeFis will take this step towards a completely compliant operational model.
Of course, most project parties won’t be able to embark on the road to securities compliance; the future crypto world will consist of both compliant and non-compliant parts, each with clear boundaries but also closely interconnected."
compliant ecosystem | non-compliant ecosystem | |
---|---|---|
Capitals | Onshore institutional capital, low-risk-preference individuals |
Offshore institutional capital, crypto-native, high-risk preference individuals |
Underlying asset | BTC, ETH, a few compliant large-cap tokens |
Most small and medium market cap tokens |
Exchanges | Licensed onshore exchange, some regulated DEXs |
Unlicensed offshore exchange, some unregulated DEXs |
Features of the Market |
Lower returns, lower volatility, safer and more transparent, more mature and stable |
Higher returns, higher volatility, more opaque and risky, more innovation and opportunities |
Complementarity | The price rise of mainstream coins and the asset appreciation will bring overflowing liquidity, which can still drive the price of small and medium-sized coins in the non-compliant ecosystem. |
A more flexible and open environment nurtures new opportunities, and as small and medium-sized coins gradually grow, some will enter the compliant ecosystem. |
Such a coexistence pattern already exists today, but the influence of the compliant ecosystem in the crypto world is still relatively small. As the regulatory framework becomes clearer, the influence and importance of the compliant ecosystem will become increasingly significant. The development of the compliant ecosystem will not only significantly increase the total scale of the entire crypto industry, but also “transfuse” a large amount of liquidity to the non-compliant ecosystem through the rise in prices of mainstream assets and resulting liquidity overflow.
💡 Large projects will become compliant, while smaller projects can remain in the non-compliant market and still enjoy the overflow of liquidity from the compliant market. The two markets will complement each other ecologically, proving that securities laws will not be the end of crypto.
On the judicial side, the SEC vs Ripple case has yet to be settled, and the SEC vs Coinbase/Binance cases have just begun - the settling of these cases could take several years.
On the legislative side, since July, several crypto regulation bills have been submitted to both houses, including the Financial Innovation and Technology for the 21st Century Act, Responsible Financial Innovation Act, Crypto-Asset National Security Enhancement and Enforcement —— Historically, more than 50 crypto-related regulatory bills have been submitted to both houses, but we are still far from a clear legal framework.
Statistics on the passing rate of bills in the US House of Representatives throughout history. On average, Congress receives about 7,000 bill submissions each year, with about 400 being enacted. https://www.govtrack.us/congress/bills/statistics
The worst outcome for the crypto industry is not that most tokens will eventually be classified as securities, but the loss of time and space for the industry to grow, and the waste of resources and opportunities, due to the long-term lack of a clear regulatory framework.
The escalation and intensification of conflicts between regulators and the crypto industry is good news, as it means that resolution is nearing.
The verdict for Ripple Labs was announced on July 13, and the next day, July 14, is the anniversary of the French Revolution. This reminds me of the unrest in France after the revolution; but it was also during that chaotic time that the foundation of modern law - the French Civil Code - was born. I hope that we can see that, although the crypto industry is currently experiencing chaos and turmoil, it will eventually find its direction and way out, establishing a set of norms and codes that can coexist harmoniously with the outside world.
Code civil des Français
📎 Phoenix Capital Management is a fundamental-driven cryptocurrency hedge fund. The founding team has held key positions in several multi-billion dollar hedge funds. We strive to use a rigorous and scientific methodology, combining top-down macro research with bottom-up industry insights, to capture structural investment opportunities in the cryptocurrency industry and create long-term returns that transcend bull and bear cycles.
You can find all our writings here: Writings .
🤩 Hiring! We are actively searching for crypto researchers to join our team. If you are interested, please send your resume to info@phoenixfund.xyz. Details can be found here.
Disclaimer:
This content is for informational use only and is not intended as financial or legal advice.
Any mistakes or delays in this information, and any resulting damages, are not the responsibility of the author. Please be aware that this information may be updated without notice.
This content does not promote or recommend the purchase or sale of any financial instruments or securities discussed.
The author may hold positions in the securities or tokens discussed in this content.
]]>Feedback Loops | Cognitive Load | Flow State | |
---|---|---|---|
People | Satisfaction with automated test speed and results Satisfaction with time it takes to validate a local change Satistaction with time it takes to deploy a change to production |
Perception of codebase complexity Ease of debugging production systems Ease of understanding documentation |
Subjective perception of staying focused and avoiding distractions Satisfaction with task or project goal clarity Perception of interruptions during on-call |
Process | Time required to generate CI results Code review turnaround time Deployment lead time (time required to release changes to production) |
Time required to get answers to technical questions Manual steps required for deploying changes Frequency of documentation improvements |
Number of time blocks without meetings or interruptions Frequency of unplanned tasks or requests Frequency of incidents requiring team attention |
Goals
Feedback loops play a vital role in software development by optimizing the value stream and reducing delays in software delivery. The faster developers receive feedback, the quicker they can make necessary adjustments and course corrections. Research indicates that frequent deployment and shorter lead times can double the likelihood of meeting performance goals.
To improve DevEx, organizations must focus on shortening feedback loops. Slow feedback not only interrupts the development process but also leads to frustration and delays. Identifying areas where tools can be optimized or human processes improved is essential for enhancing the feedback loop process.
Cognitive load refers to the mental processing required by a developer to perform a task. As the number of tools and technologies grows, developers face an increasing cognitive load, which can sometimes hamper their ability to deliver value.
High cognitive load can arise due to issues such as poorly documented code or complex development processes. To improve DevEx, organizations should eliminate unnecessary hurdles in the development process. This includes emphasizing organized code and documentation, as well as providing easy-to-use, self-service tools that facilitate a smoother workflow.
Flow state is a mental state characterized by full immersion, energized focus, and enjoyment in an activity. Developers often describe this state as “getting into the flow” or “being in the zone.” Achieving a flow state leads to higher productivity, innovation, and employee development.
Studies have shown that developers who enjoy their work and frequently experience the flow state perform better and produce higher-quality products. However, delays and interruptions can hinder developers from reaching this productive state.
To enhance DevEx, organizations should focus on creating optimal conditions for the flow state. This includes minimizing disruptions by clustering meetings, avoiding unplanned work, and batching help requests. Additionally, fostering a positive team culture that gives developers autonomy and opportunities to work on fulfilling challenges is crucial for facilitating flow state. Leaders should promote environments conducive to these conditions.
By focusing on the three core dimensions of DevEx - feedback loops, cognitive load, and flow state - organizations can better understand and improve developer productivity. By optimizing these areas, teams can experience significant improvements in their output, ultimately leading to more successful delivery of software.
]]>Optimism is an EVM equivalent, optimistic rollup protocol designed to scale Ethereum.
Optimistic rollup works by bundling multiple transactions into a single transaction, which is then verified by a smart contract on the Ethereum network. This process is called “rolling up” because the individual transactions are combined into a larger transaction that is submitted to the Ethereum network. The term “optimistic” refers to the fact that the system assumes that transactions are valid unless proven otherwise, which allows for faster and more efficient processing of transactions.
The rollup node can run either in validator or sequencer mode:
The batch submitter, also referred to as the batcher, is the entity submitting the L2 sequencer data to L1, to make it available for verifiers.
Proposer generates and submitting L2 Output checkpoints to the L2 output oracle contract on Ethereum. After finalization period has passed, this data enables withdrawals.
Both batcher and proposer submit states to L1. Why are they separated?
Batcher collect and submit tx data into L1 with a batch, while proposer submits the commitments (output roots) to the L2’s state, which finalizes the view of L2 account states. They are decoupled so that they can work in parallel for efficiency.
Various contracts for L2 to interact with the L1:
Understanding the OP stack can be challenging due to a number of factors. One such factor is the numerous components that are referred to multiple times with slightly different names in code and documentation. For example, the terms “op-batcher” and “batch-submitter” / “verifiers” and “validators” may be used interchangeably, leading to confusion and difficulty in understanding the exact function of each component.
Another challenge in understanding the OP stack is the evolving architecture, which may result in some design elements becoming deprecated over time. Unfortunately, the documentation may not always be updated to reflect these changes. This can lead to further confusion and difficulty in understanding the system, as users may be working with outdated or inaccurate information.
To overcome these challenges, it is important to carefully review all available documentation, to keep concepts consistently across places, and to stay up-to-date with any changes or updates to the OP stack. This may require additional research and collaboration with other users or developers, but it is essential in order to fully understand and effectively utilize this complex system.
]]>Token steaming means sending recurring payments in real time, like water flowing into its target. There are two kinds of payment innovations:
Projects | Blockchains | Payout | Accept Payments | Differentiation |
---|---|---|---|---|
Sablier | EVM | ✅ | ❌ | protocol for real-time finance, protocol + app |
Superfluid | EVM | ✅ | ❌ | stream money every second, protocol + app |
Roke.to | NEAR | ✅ | ❌ | stream money, protocol + app |
Zebec | Solana | ✅ | ✅ | multisig treasury management and streaming payments |
Streamflow | Solana | ✅ | ✅ | token distribution platform, token vesting and payroll |
MeanFi | Solana | ✅ | ❌ | manage Your Treasury With Real-Time Finance |
calamus.finance | Multi-chain | ✅ | ❌ | real-time payment and token vesting |
llamapay | EVM | ✅ | ❌ | automate transactions and stream them by the second. salary, vesting, payments. |
Suberra | EVM | ❌ | ✅ | accept crypto for commerce, one-time payments or recurring subscriptions |
LoopCrypto | Ethereum, Polygon | ❌ | ✅ | payment links, receipts and reminders, dashboard, web hooks |
diagonal.finance | EVM | ❌ | ✅ | non-custodial - Multiple models fixed, seat, usage-based, or Superfluid streaming |
radom.network | NEAR, Aurora | ✅ | ❌ | pay web2 services with crypto |
spritz.finance | EVM | ✅ | ❌ | pay bills with crypto |
cask.fi | EVM | ✅ | ✅ | non-custodial protocol for auto payment |
DataMynt | Multi-chain | ✅ | ✅ | for business, deposit, settlement, payment, invoice |
Orbital | Multi-chain | ✅ | ✅ | web2 + web3 corporate financial services |
Coinbase commerce | Multi-chain | ❌ | ✅ | merchants accept payments with custodial and non-custodial wallets and allow customers to checkout |
wink.finance | Multi-chain | ✅ | ❌ | simplifies payments and expense management, multisig |
As of the end of 2022, the most prominent web3 payment protocol is probably EIP-86/EIP-4337 for Account Abstraction. It uses smart-contract wallets to decouple private key ownership from asset account ownership. The protocol is still a work in progress on Ethereum, but Visa has implemented auto payments for self-custodial wallets on Starkware in its internal hackathon.
]]>The first thing you need to know about a system design interview is that you have to be talkative throughout the interview session. Of course, you must consult the interviewer to determine whether you are on the right track to give them what they want; however, you still need to prove that you can do the job independently. So, ideally, keep talking about what the interviewer expects throughout the interview before they even have to ask.
Secondly, do not limit yourself to only one solution. Given the same problem, there could be so many ways to solve it that it takes no license to be an engineer. There are pros and cons to all the choices you will make. Discuss tradeoffs with your interviewer, and pick the most suitable solution to your assumptions and constraints. It’s like, in the real world, people won’t build the Golden Gate bridge over a trench, nor will they build a temporary bridge over San Francisco Bay.
Finally, to excel in the interview, you’d better bring something new. “Good engineers script; great engineers innovate”. If you cannot teach people something new, you are just good, not great. Quality answers = Novelty x Resonance.
If you are not sure how to navigate the session and be talkative all the time, here is a simple 4-step template you can follow in a divide-and-conquer way:
All the designs in this book will follow these steps.
Specifically for this “Design Pinterest”, I will explain everything as detailed as possible because it is the first case of the entire book. However, for simplicity, I won’t cover many of the elements here in other designs of this book.
All systems exist for a purpose, so with software ones. Meanwhile, software engineers are not artists - we build stuff to fulfill customers’ needs. Thus, we should always start with the customer. Meanwhile, to fit the design into a 45-minute session, we must set constraints and scope the work by making assumptions.
Pinterest is a highly scalable photo-sharing service with hundreds of millions of monthly active users. Here are the requirements:
Do not dive into details before outlining the big picture. Otherwise, going off too far in the wrong direction would waste time and prevent you from finishing the task.
Here is the high-level architecture, in which arrows indicate dependencies. (Sometimes, people would use arrows to describe the direction of data flow.)
Once the archiecture is there, we could confirm with the interviewer if they want to go through each component with you. Sometimes, the interviewer may want to zoom into an unexpected domain problem like designing a photo store (that’s why I am always saying there is no one-size-fits-all system design solution. Keep learning…). However, here, let’s still assume that we are building the core abstraction: upload a photo and then publish to followers.
Again, I will explain as much as possible in a top-down order because this is our first design example. In the real world, you don’t have to go through each component in such a level of detail literally; instead, you should focus on the core abstraction first.
Mobile and browser clients connect to the Pinterest data center via edge servers. An edge server is an edge device that provides an entry point into a network. Here we see two kinds of edge servers in the diagram - load balancers and reverse proxy.
Load balancers distribute incoming network traffic to a group of backend servers. They fall into three categories:
A load balancer could exist in many other places as long as there is a need for balancing traffic.
Unlike a “forward” proxy in front of clients that route traffic to an external network, a reverse proxy is a kind of proxy sitting in front of servers, so it’s called “reverse”. By this definition, a load balancer is also a reverse proxy.
Reverse proxy brings a lot of benefits according to how you use it, and here are some typical ones:
Nginx, Varnish, HAProxy, and AWS Elastic Load balancing are popular products on the market. I find it handy but powerful to write a lightweight reverse proxy in Golang. In the context of Kubernetes, it’s basically what Ingress and Ingress Controllers are doing.
This is where we serve web pages. In the early days, web service usually combines the backend with page rendering, as Django and Ruby on Rails frameworks do. Later, growing with the project size, they are often decoupled to dedicated fronend and backend projects. Frontend focuses on App rendering while the backend serves the APIs for the frontend to consume.
Most backend engineers are not familiar with mobile design patterns, go to iOS Architecture Patterns for more.
A dedicated frontend web project is very similar to a standalone mobile app - they are both clients of the servers. Some people would call them “holistic frontend”, when engineers can build user experiences on both platforms simultaneously, like react for web and react-native for mobile.
Clients talk to the servers via public APIs. Nowadays, people often serve RESTful or GraphQL APIs. Learn more in public API choices.
There are two major bottlenecks of the whole system – load (requests per second) and bandwidth. We could improve the situation
Internet companies prefer scaling out, since
To scale out, we’d better keep services stateless, meaning they don’t hold states in local memory or storage, so we could kill them unexpectedly or restart them anytime for any reason.
Learn more about scaling in how to scale a web service.
The single responsibility principle advocates small and autonomous services that work together so that Each service can “do one thing and do it well”, and grow independently. Small teams owning small services can plan much more aggressively for hyper-growth. Learn more about Micro Services vs. Monolithic Services in Designing Uber
How do those services find each other?
Zookeeper is a popular and centralized choice. Instances with name, address, port, etc. are registered into the path in ZooKeeper for each service. If one service does not know where to find another service, it can query Zookeeper for the location and memorize it until that location is unavailable.
Zookeeper is a CP system in terms of CAP theorem (See Section 2.3 for more discussion), which means it stays consistent in the case of failures, but the leader of the centralized consensus will be unavailable for registering new services.
In contrast to Zookeeper, Uber did some interesting work in a decentralized way, named hyperbahn, based on Ringpop consistent hash ring, though it turned out to be a big failure. Read Amazon’s Dynamo to understand AP and eventual consistency.
In the context of Kubernetes, I would like to use service objects and Kube-proxy, so it would be easy for programmers to specify the address of the target service with internal DNS.
The follower-and-followee relationship is all around these two straightforward data structures:
Map<Followee, List of Followers>
Map<Follower, List of Followees>
A key-value store, like Redis, is very suitable here because the data structure is pretty simple, and this service should be mission-critical with high performance and low latency.
The follower service serves functionalities for followers and followees. For an image to appear in the feed, there are two models to make it happen.
Map <Followee, List of Followers>
fan-out is too large, then the push model will cost a lot of time and data duplicates.Map<Follower, List of Followees>
fan-out is too large, then the pull model will spend a lot of time iterating the huge followee list.The feed service stores the image post metadata like URL, name, description, location, etc, in a database, while images themselves are usually saved in a Blob Storage like AWS S3 and Azure Blob store. Take S3 for example, a possible solution is like the following when the customer creates a post with the web or mobile client:
Customers post to feeds as time passes, so HBase / Cassandra’s timestamp index is an excellent fit for this use case.
Transmitting blobs consumes a lot of brandwiths. Once we uploaded the blob, we read them a lot but seldemly update or delete it. Thus, developers often cache them with CDNs which will distribute those blobs to a closer place to the customer.
AWS CloudFront CDN + S3 might be the most popular combination on the market. I personally use BunnyCDN for my online content. Web3 developers like to use a decentralized store like IPFS and Arware.
The search service connects to all the possible data sources and index them so that people could easily search feeds. We usually use ElasticSearch or Algolia to do the work.
The spam service uses machine learning techniques like supervised and unsupervised learning to mark and delete profanity content and fake accounts. Learn more in Fraud Detection with Semi-supervised Learning.
What are the blindspots or bottlenecks of the design above?
There are two directions that we could approach the estimation problem: top-down and bottom-up.
For bottom-up, you do load tests with the existing system and plan the future on the company’s current performance and future growth rate.
For top-down, you start with the customers in theory and make the back-of-the-envelope calculation. I highly recommend you do it with a digital spreadsheet, where you can easily list the formula and the assumed/calcuated numbers.
When we rely on external blob storage and CDN, bandwidth is unlikely to be a problem. So I will estimate the capacity for the follower service as an example:
Row | Description ("/" means per) | Estimated Number | Calculated |
---|---|---|---|
A | daily active users | 33,000,000 | |
B | requests / user / day | 60 | |
C | rps / machine | 10,000 (c10k problem) | |
D | scale factor (redundency for user growth in 1 yr) |
3 times | |
E | Number of service instances | = A * B / (24 * 3600) / C * D | ~= 7 |
We can see that Row E is a calculated result of the formula. After applying this estimation method to each one of those microservices and storages, we will better understand the entire system.
Real-world capacity planning is not a one-time deal. Provisioning too many machines will waste money, and preparing too few ones will cause outages. We usually do it with a few cycles of estimation and experimentation to find the right answer; or use autoscaling if the system supports this and budgets are not a problem.
Big corp engineers are often indulged with abundant computing and storage resources. However, great engineers will think about costs and benefits. I would sometimes experiment with different tiers of machines and add rows for their monthly expenses for estimation.
]]>In Zanzibar’s context, we can express the AuthZ question in this way:
isAuthorized(user, relation, object) = does the user have relation to object?
It’s called relationship-based access control (ReBAC). Clients could build ABAC and RBAC on top of ReBAC. Unfortunately, Zanzibar is not open-sourced nor purchasable as a out-of-box service.
Zanzibar Architecture
Why is Zanzibar scalable?
Auth0 FGA is an open-source implementation of Google Zanzibar. Check the interactive tutorial at https://zanzibar.academy/.
For enterprise developers in the context of microservices, how to use the managed solution of FGA?
Unfortunately, I don’t see changelog audits and version control to rollback in case developers break things in the FGA dashboard, probably because FGA is still a work in progress.
With Oso, you can:
Keto is an open Source (Go) implementation of Zanzibar. Ships gRPC, REST APIs, newSQL, and an easy and granular permission language (DSL). Supports ACL, RBAC, and other access models.
SpiceDB is an open-source database system for managing security-critical application permissions inspired by Google’s Zanzibar paper.
Topaz is an open-source authorization service providing fine-grained, real-time, policy-based access control for applications and APIs.
It uses the Open Policy Agent (OPA) as its decision engine, and provides a built-in directory that is inspired by the Google Zanzibar data model.
Authorization policies can leverage user attributes, group membership, application resources, and relationships between them. All data used for authorization is modeled and stored locally in an embedded database, so authorization decisions can be evaluated quickly and efficiently.
It seems to be an integrated CIAM solution, and there is no standalone feature for enterprise authorization. Documentation is confusing…
The Open Policy Agent (OPA) is an open-source, general-purpose policy engine that unifies policy enforcement across the stack. OPA provides a high-level declarative language that lets you specify policy as code and simple APIs to offload policy decision-making from your software. You can use OPA to enforce policies in microservices, Kubernetes, CI/CD pipelines, API gateways, and more.
OPA was originally created by Styra and a graduated project from Cloud Native Computing Foundation (CNCF).
Permit.IO is a low-code AuthZ platform based on OPA and OPAL.
Scaled Access is an european company that was acquired by onewelcome. It offers rich context-aware access control, real-time policy enforcement, fine-grained authorization, and relationship-based access control. There are APIs in the documentation but no SDKs.
Casbin is an authorization library that supports access control models like ACL, RBAC, ABAC in Golang. There are SDKs in many programming languages. However, its configuration is pretty static in CSV files, and it’s more for corporation internal and less for customer-facing authorization.
This service looks pretty scrappy - beautiful websites without any content for developers. No doc, no video or self-service demo. I suspect its positioning is for non-tech enterprises. Not recommended.
Here is a preliminary ranking after my initial check. Ideally, I want a LaunchDarkly-like AuthZ platform - easy to integrate and operate, fully equipped with audit logs, version control, and a developer-facing web portal.
Github Stars | Models | DevEx | Perf | Score (out of 5) | |
---|---|---|---|---|---|
Oso | 2.8k | ReBAC | DSL, API, SDK, web portal | ? | 3 |
Spicedb | 3k | ReBAC | DSL, API, SDK, web portal | ? | 3 |
permit.io | 840 | ReBAC | DSL, API, SDK, low-code web portal | ? | 3 |
Aserto Topas | 534 | ReBAC | DSL, API, SDK, web portal | ? | 3 |
FGA | 657 | ReBAC | DSL, API, SDK, web portal | ? | 3 |
Keto | 3.8k | ReBAC | DSL, API, SDK | ? | 2 |
Casbin | 13.4k | ABAC, RBAC | Library, static file for policies | ? | 1 |
Improving docs is high-leverage work.
Understand your developer personas.
Developer’s questions are useful clues to improve your doc.
Segment by skill level - seeing the table of skill levels to what they need:
skill levels | Docs | Tooling | Tutorials / Examples |
---|---|---|---|
Beginner | Quickstarts | SDK, simple libs | Frontend code, zero-to-hero video series |
Intermediate | Reference Docs, topic-sorted guides | type annotations, dev console/studio | Full-fledged example apps |
Advanced | A library of ideas we’d like to see built | mini accelerator / grants | Primitives as building blocks for larger apps |
Segment by role / intent
Role | Value |
---|---|
hackathon / indie dev | mediate |
dev from an integration partner | mediate |
future founder | high |
That’s why I am always respectful towards all kinds of innovations - no matter how small it appears today, who would imagine it will take over the world tens of years later?
As a builder or businessman, we have a product or service to sell, and the question is - how to speed up the process for it to take over the market?
Everett M. Rogers came up with Five Intrinsic Attributes of Innovation in his Diffusion of Innovation Theory :
In addition to the intrinsic attributes above, there are interactions between the innovation and the market segments. We call it the technology adoption lifecycle (TAL), which categorizes customers on the market into five pieces.
The Chasm theory indicates that there is no smooth transition from early adopters to the early majority because those two market segments want different value propositions. Crossing the Chasm applies the “D-Day analogy” to solve the problem - focus, focus, focus! Focus is all it takes to attack each segment one by one - like D-Day - you take over the beach first and then move to the next target.
Every entrepreneur would dream of a beautiful S-curve for their innovation to diffuse into the market. So to unveil the math curtain, let’s see how Scott Page explains it in the book model thinker, Chapter 11: Broadcast, Diffusion, and Contagion.
The abstraction here is to partition the population into two groups:
Group informed starts empty, and group susceptible is all the relevant population exposed to conversion. The growth curves are in various shapes with various models to convert people from susceptible to informed.
This model assumes that
And then we get this formula
We could learn from the model that…
This model assumes that
And then we get this formula
We could learn from the model that…
Most consumer goods and info spread through both broadcast and diffusion. Usually, for the same product, companies are running ad campaigns; meanwhile, customers are referring new customers.
All the models above assume no moving back from informed to susceptible. We seldomly abandon our adoption of many home appliances - dishwashers, air dryers, etc. However, it is not the same for fashion styles, diseases, and … your brand in the real world. In this case, things are contagious only for a particular time. People may forget your product as time passes by and then get recovered.
Let’s introduce the probability of recovery, then we get the susceptible-infected-recovered (SIR) model.
For disease control, the infected will rise first, and we hope it will eventually drop.
However, we hope the informed will rise to the top for our products. The SIR model produces a tipping point, aka, basic reproduction number ().
Products with spread through the population, while products with dissipate.
Take COVID as an example. Its is 2 to 3. and that is why people wear masks, keep distances from others and avoid crowds to lower the diffuse probabilities.
is the ultimate question for marketing - would your marketing be contagious enough to fight against forgetfulness?
By the formal definition, the mass media version of the tipping point is usually wrong. For example, a kink is not a tipping point in the chart below for the number of Google Plus users in the first 14 days. Instead, is the real tipping point.
Finally, here is the summarization of all components that are worth optimizing for go-to-market campaigns. Please note that spreading information carries costs, unlike diseases, so we need to consider ROI.
## 20XX-XX-XX Company Name Investment Memo
| Attribute | Value |
| -------------------- | ----- |
| Category | |
| Round | |
| Raising | |
| Pre-money Valuation | |
| Post-money Valuation | |
| Allocation | |
## Summary
The decision is yes / no with an amount of X, because of the most significant argument Z.
- highlight 1, could be pros and cons.
- highlight 2
- highlight 3
Ratings: X out of 5 (benchmark against past deals)
| Attribute | Value |
| -------------------- | ----- |
| Traction | |
| Team | |
| Product | |
| Social Proof | |
| Pitch / Presentation | |
| Total | |
## Introduction
- What does the company do?
- What is the problem the company solving?
- How does the world work now in relation to this problem?
- How does the company solve the problem?
- How does solving the problem change behavior and make money?
- What is the scale of the opportunity?
## Traction / Metrics
- Discuss traction up to now (include a chart).
- Discuss main related metrics, such as churn, ACV, rake.
- Discuss revenue drivers.
- What does the go-to-market look like?
## Challenges to Growth
- What's prevented you from growing even faster?
- How will raising money solve this problem?
## Market
- Who are the customers?
- How do those customers think / act?
- How big is the opportunity these customers represent?
## Future States
- What happens to the market as the company starts to win?
- How does the company change the market and where does that lead the company?
## Compatitive Landscape
- What is the competitive landscape and how does the company defeat it?
## Team
- Who are the team and what makes the team special?
## FAQ
- The main objections the company is likely to face, and eloquently knock them down. Data is good here.
- This is probably the part where the memo is most powerful relative to a deck.
## Use of funds
- How much have the company raised in the past?
- How much the company is raising and what are they going to do with it?
- [ ] sort qualitatively
- [ ] apply filtering criteria
- [ ] create market map
- [ ] assess risks at each life stage (TAL)
- [ ] quantify uncertainties
| Stage | Early Stage Success | Cross Chasm | Mass Market Success | Mass Market Share |
|-------------|---------------------|-------------|---------------------|-------------------|
| Market | | | | |
| Product | | | | |
| Team | | | | |
| Financial | | | | |
| Total | | | | |
- [ ] perform sensitivity analysis
- [ ] calculate risk / return
]]>20XX Yearly OKRs
* Core Objective 1:
* Core Objective 2:
* Core Objective 3:
* Stretch Objective 1:
* Stretch Objective 2:
20XXQX OKRs
* Core Objective 1:
* KR:
* KR:
* Core Objective 2:
* Stretch Objective 1:
]]># [short title of solved problem and solution]
* Status: [proposed | rejected | accepted | deprecated | … | superseded by [ADR-0005](0005-example.md)] <!-- optional -->
* Deciders: [list everyone involved in the decision] <!-- optional -->
* Date: [YYYY-MM-DD when the decision was last updated] <!-- optional -->
Technical Story: [description | ticket/issue URL] <!-- optional -->
## Context and Problem Statement
[Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question.]
## Decision Drivers <!-- optional -->
* [driver 1, e.g., a force, facing concern, …]
* [driver 2, e.g., a force, facing concern, …]
* … <!-- numbers of drivers can vary -->
## Considered Options
* [option 1]
* [option 2]
* [option 3]
* … <!-- numbers of options can vary -->
## Decision Outcome
Chosen option: "[option 1]", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | … | comes out best (see below)].
### Positive Consequences <!-- optional -->
* [e.g., improvement of quality attribute satisfaction, follow-up decisions required, …]
* …
### Negative Consequences <!-- optional -->
* [e.g., compromising quality attribute, follow-up decisions required, …]
* …
## Pros and Cons of the Options <!-- optional -->
### [option 1]
[example | description | pointer to more information | …] <!-- optional -->
* Good, because [argument a]
* Good, because [argument b]
* Bad, because [argument c]
* … <!-- numbers of pros and cons can vary -->
### [option 2]
[example | description | pointer to more information | …] <!-- optional -->
* Good, because [argument a]
* Good, because [argument b]
* Bad, because [argument c]
* … <!-- numbers of pros and cons can vary -->
### [option 3]
[example | description | pointer to more information | …] <!-- optional -->
* Good, because [argument a]
* Good, because [argument b]
* Bad, because [argument c]
* … <!-- numbers of pros and cons can vary -->
## Links <!-- optional -->
* [Link type] [Link to ADR] <!-- example: Refined by [ADR-0005](0005-example.md) -->
* … <!-- numbers of links can vary -->
]]>Why does it matter?
Copied and modified from this article.
Subheading: One sentence saying who the market is and what the benefit is
Summary: 2–4 sentences that gives a summary of the product and the benefits. Should start with customer and be self-contained so that a person could read only this paragraph and still understand the new product/feature.
Problem : 2–4 sentences describing the problem that a customer faces, which this product solves. Tests your assumptions about the pain-points that you are addressing.
Solution : 2–4 sentences, describing how the new product/feature addresses this problem. Tests your assumptions about how you are solving the pain-points.
Getting started: 1–3 sentences describing how someone can start using this product/feature (if it’s baked into the existing product, say this explicitly). Tests your assumptions about how easy the ramp-up is for your customers to take advantage of the new product/feature.
Internal quote: Someone within your company being quoted about what they like about the product/feature. Tests your assumptions about the value you are creating for your customers and how you position this product within your broader product offerings.
Customer Quote(s): a hypothetical customer saying what they like about the new product/feature. Tests your assumptions about how you want your customers to react to the new product/feature and your ideal customer profile. They should be doing something that they couldn’t do before, doing something much quicker and easier, saving time and effort, or in some other way making their life better. Whatever the benefit is, their delight in the benefit(s) should be exhibited in the quote. This should be multiple quotes from different customers if you have multiple profiles of ideal customers, example: mid-market and F50 customers.
Call to action: 1–2 sentences telling the reader where they can go next to start using the product/feature. Tests your assumptions about whether this is a feature that is automatically on, something they need to turn on, a beta-release, etc.
A set of public frequently-asked questions and their answers. This should be a comprehensive list of everything that a customer might want to know about the product. It should include any reasonable question that comes up when discussing the new product/feature with customers and customer-facing teams during the development of the product/feature.
A set of private, internal frequently-asked questions and their answers in a format that can be understood by every other stakeholder. An FAQ might include wireframes of a product with a strong UX component, or a link to separate wireframe documents, but the PR should rely on text alone. This will allow all internal stakeholders to get clarity on the product/feature.
Prediction | Validation | |
---|---|---|
Item 1 | ||
Item 2 | ||
Item 3 |
Dsruptive innovation vs. Continuous innovation
High-tech industries introduce disruptive innovation routinely, during which people are converted into customers by following a pattern of normal distribution. The product’s user growth follows an S-curve.
Disruptive innovation’s customers are converted at different stages in the technology adoption life cycle. They are…
Segment | What They Want |
---|---|
Innovators | novel, cool and experimental things |
Early Adopters | gaining advantages or getting products before others |
Early Majority | proven ROI, instant access, low transition costs, support available |
Late Majority | adopting as minimal as possible or only when everyone else has adopted |
Laggards | avoidance to adopt new things |
This cycle provides guidance of the high tech marketing model: the way to develop a high-tech market is to work the curve left to right, focusing on each group one by one, because groups on the left promote products for the right ones in a momentum.
Momentum is vital because it can
Inspecting into the technology adoption lifecycle, we can see
two cracks
and one CHASM
Early adopter-to-majority chasm. Because their needs are different
The compatibility above leads to two key points
Who did fall into the early adopter-to-majority chasm in 2014? E.g., holograms, pen-based tablets, fuel cells, QR codes (in the US), Massive Open Online Courses, Segways, Motorola iridium.
Internal
Beancount is a computer language that enables double-entry bookkeeping in text files. Once you define financial transactions in the file, it will generate various reports. Martin Blais, the designer of this language, argues that command-line bookkeeping has many advantages - It is fast, portable, open, and customized.
We strongly agree with the argument and share the feeling of empowerment brought by beancount language. And we wanted to do more - introducing the technology to more people. It means that we have to improve the usability and make it more accessible to a broader audience.
Not everyone is a command-line enthusiast, and this is why we build Beancount.io - the personal finance manager for everyone. Here is how it works:
For heavy-duty work, beancounters could still use their computers to edit or view the ledger with their browsers visiting https://beancount.io or syncing with Dropbox. This keeps the flexibility of the command-line tools, while not losing the cross-device access of the cloud-based solution.
For daily light-weight operations, such as instantly adding an entry, beancounters could use the mobile app to connect to the secured cloud.
Mike Thrift, a backend engineering working on this product, says
I used to set up a reminder every day for myself to open my laptop and input records to my bean files. Now, with beancount.io, it is way easier for me to modify my ledger whenever I need it, even when I am outdoors purchasing something in the store.
Zhi Li, a software engineer from Facebook, tells us
I have migrated all my beancount files to beancount.io, and now it works perfectly for my day-to-day usage. I have paid for Pro features like automatic data backup, but I feel there are more things you guys could do to improve the service.
You could sign up now at https://beancount.io/sign-up/ or download iOS or Android App. We streamlined the registration to collect as minimal information as we can from you to bootstrap the service. Then you will get a preset empty ledger that is ready for you to add an entry right away.
The great plateau is the career state that the maintenance of the status quo consumes all your time and energy so that you cannot break through and reach the next level of your life.
Here is some advice to help you move forward.
What are the factors that are pumping or dumping you? Are those factors serving your goals instead of contradicting your goals? Are those goals aligning with each other instead of violating each other?
Be realistic and refer to the base rate ratio when analyzing and setting the goal. For example, Jeff Bezos told a story about a handstand coach’s experience - most people believe that they can learn handstand in two weeks; however, it usually takes six months. When you get stuck, the answers are more likely to be from others, from reality, not from yourself.
Stop aiming at a moving target. Your current situation is possibly what you desired four or five years ago. Don’t be too greedy :)
More specific to those dumping you and wasting your time and energy, could we remove those costs? Maybe that will hurt the upside temporarily; however, could that be the hockey stick growth afterward?
Take a sheet to list your daily operations and mark them as burdens or not. If yes, how to remove them?
To shake yourself to leave the local optima, you need to set aside some time specifically for something new. The process seems useless at the beginning and may take long time.
You have to be patient. Like what Steve Jobs did when he returned to Apple after the exile, he cut less profitable product lines and waited for the next big wave.
Business is an infinite game, and you can always accumulate comparative advantages over time. If you have some extra time and are not sure of a clear goal for now, you could always invest in yourself - better mental and physical health to help you fight through battles in the future. Keep learning and knowing more to increase the probability of success. Optimize the business to work more efficiently and live longer on the market.
Finally, do not underestimate your growth. There is a deception phase even with prominent technologies like AI or 3D printing; they seem not to progress for a long time and then suddenly improve at an exponential rate.
]]>We, engineers, often boast about leadership without a clear definition of what we are saying. We boast with authority - X years of experience, intimidating titles from prestigious companies, and quotes from big names. We boast with emotions - close friends achieving financial freedom, leaders making a huge impact, and how exciting business is taking off. We boast with logic - the team should be united, we are a team, and then we should be united.
It is OK if boasting is a personal matter. However, unfortunately, flattery lives in the nature of hierarchical corporate life, especially for those from East Asia. Meanwhile, people in the United States tend to say good things to each other so that both parties can feel better. Praises are literally everywhere.
As a result, leadership is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it.
Here are the definitions to distill the clarity from those chaotic ramblings of the mass.
What is leadership?
Leadership is organizing people to solve challenging problems.
What does it really mean?
What is technology?
Technology is a scientific way of solving problems.
What does it really mean?
What is innovation?
Innovation is creating something new.
What does it really mean?
Well, what is technology leadership?
]]>Technology leadership is organizing people to solve challenging problems with technologies, usually through innovations.
Decisiveness: strategy, tactics, and making fact-based decisions faster. Play progressively from small wins to more significant wins for the individuals and organizations. Gathering information, diagnosing to gain insights, making guiding policies, and taking coherent actions. Differentiating facts from opinions and making sure that options are believability-weighted. Exploring innovations in the entire process, and innovation means making changes to do things 10x better. Making initial decisions fast within the time window, reflecting on results, and then iterating through the hierarchy of lifetime/years/quarters/weeks.
Execution Engine: Building the system to deliver more and then in higher quality, instead of being proud of a few deliverables. Attention to detail. Nudging the cross-functional teams to your unique vision. Balancing speed, quality, and scope according to the customer’s requirements.
Domain Expertise: pursuing un-teachable special know-how. been there/done that. The market rewards uniqueness.
Product-sense: developing the intuition on what a great product is. Perceiving the market and industry with data. Building and operating the product with proper processes. Synthesize with other elements in the big picture. Invent and simplify. Plan and manage change. Customer-obsession.
People and Culture: Work with others: shaping the world together with people. Lead by example. Teamwork. Making people happy and then productive. Sort people and projects often / be professionally judgemental. Incentivize people with intrinsic and then extrinsic motivations. Orgnization = People + Culture. Being both capable and warm. Telling a story that inspires your own passion first. Empowering people to achieve more. Self-driven. Listen to people and inspire people to share. hiring and coaching team. humility, low ego. best idea wins. growth mindset.
Synergy & Resourcefulness: aligning or connecting resources. 1 + 1 > 2. optimizing the web of customers, distribution channels, products, people, technology, and capitals. playing the reputation game in a long term. turning more people into stake owners and aligning to shared goals.
First-things-first, let’s get back to basics
In the beginning… Let there be a simple service…
Then, as the business grows, we scale the system with AKF scale cube:
Plus Conway’s law: organization designs the systems mirroring its communication structure. We usually evolve the architecture to micro-services (see why microservices? for more)
In the microservice world, let’s take a functional slice of the authn and authz services, and there is an Identity and Access Management (IAM) team working on it.
Identity Provider
Workflow: User Settings and Profile Updates
Ory.sh/Kratos as an Example Architecture
OAuth2 let the user or client go through four major workflows (not sure which one to use? see this) like
And then finally get the access token and refresh token
The assumption is that there are so many entities involved in this workflow - client, resource owner, authorization server, resource server, network, etc. More entities introduce more exposure to attack. A comprehensive protocol should consider all kinds of edge cases. For example, what if the network is not HTTPs / cannot be fully trusted?
OpenID connect is the identity protocol based on OAuth2, and it defines customizable RESTful API for products to implement Single Sign-On (SSO).
There are a lot of tricky details in those workflows and token handling processes. Don’t reinvent the wheel.
Problem: Credential stuffing attack
Users tend to reuse the same username and password across multiple sites. When one of those sites suffers from a data breach, hackers brute-force attack other sites with those leaked credentials.
Challenge: Bad deliverability of Email or SMS
How could clients subscribe to the server’s state? Short polling, long polling, web socket, or server-sent events.
Don’t reinvent the wheel.
Challenge 1: Web login is super slow or cannot submit login form at all.
See Web App Delivery Optimization for more info
Challenge 2: Account taking-over
Challenge 3: Account creation takes too long
When the backend system gets too large, a user creation may fan out to many services and create a lot of entries in different data sources. It feels bad to wait for 15 seconds at the end of sign up, right?
isAuthorized(subject, action, resource)
{
"subjects": ["alice"],
"resources": ["blog_posts:my-first-blog-post"],
"actions": ["delete"],
"effect": "allow"
}
Challenge: single point of failure and cascading failures
Western culture has a tradition to respect privacy, especially after the Nazis murdered millions of people. Here are some typical sensitive data types: Personally Identifiable Information (PII), Protected Health Information (PHI, regulated by HIPAA), and Credit Card or Payment Card Industry (PCI) Information.
Redacting sensitive information alone may not be good enough to prevent data associated with other datasets.
Differential privacy helps analysts extract data from the databases containing personal information but still protects individuals’ privacy.
To decouple id from a centralized identity provider and its associated sensitive data, we can use decentralized id (DID) instead.
did:example:123456789abcdefghijk
Imagine that Alice has a state-issued DID and wants to buy some alcohol without disclosing her real name and precise age.
A DID solution:
did:ebfeb1f712ebc6f1c276e12ec21
, name, avatar url, birthday and other sensitive data.did:ebfeb1f712ebc6f1c276e12ec21
is over the age 21This article is an overview of authn and authz in microservices, and you don’t have to memorize everything to be an expert. Here are some takeaways:
BOZ is the acronym for a big loop that engages three small loops:
Stress is a good thing for people while the distress is not.
The assumption here is that too many projects happen in parallel, and it is easy to lose focus. We should rely more on the team member mutually driving each other than one person as the single point of failure.
RACI is an acronym for the model:
RACI is an acronym for the model:
And the major challenges for the scaling process are
To enjoy a life of building software, media, and community as a hobby (all things here are NOT related to my job) / for pure pleasure - why build personal infrastructure? And what are the strategies and executions to grow hobby projects? What is my current progress?
Playing Chinese copy of Tamiya mini 4WD
And play computer games on DOS.
Introduction to programming with Macromedia Authorware
And then you can play in the classroom behind piles of textbooks :)
Lego Robots | Challenge |
---|---|
… the fascination of fashioning complex puzzle-like objects of interlocking moving parts and watching them work in subtle cycles, playing out the consequences of principles built in from the beginning.
— The Mythical Man-month
Meanwhile, I came across some mind-blowing articles.
I conclude:
Plus, some take-away from my previous pre-PMF startup experience
Here is the architecture of my hobby projects.
Technologies: React, React Native Expo, GraphQL, KOA, TypeScript, AVA, Webpack, Airflow, MongoDB, Python Pandas and Flask, svelte, Metabase, Golang, etc.
Servers and APIs: Heroku, DigitalOcean, Azure, AWS, Github Pages, BunnyCDN.
Being an early majority to adopt proven new tech
Focus on building, not wasting time on SRE
Example 1
Example 2
“This architecture is not future-proof! / does not scale!”
Well…
Living a balanced life and keep everything on track, measured by data
The key metrics for a “retention-first growth” is cohort analysis.
Benchmarks for reference:
Industry | Day 1 | Day 7 | Day 30 |
---|---|---|---|
2C | 40 | 20 | 10 |
E-commerce | 35 | 15 | 5 |
Gaming | 30 | 15 | <5 |
EdTech | 25 | 10 | 5 |
Products
Framework
Helped my friends’ projects to start from scratch
Media:
Community:
👍 Definitely and welcome! They are mostly open sourced or open for registration. Thank you for becoming our valued customer or community member!
👏 Feedback is highly appreciated!
❤️ Like it? Check this article at https://tianpan.co and follow me on https://twitter.com/intent/follow?original_referer=https%3A%2F%2Ftianpan.co%2F®ion=follow_link&screen_name=tianpan10x :)
]]>An online judge is primarily a place where you can execute code remotely for educational or recruitment purposes. In this design, we focus on designing an OJ for interview preparation like Leetcode, with the following requirements:
The architecture below is featured on queueing for async execution and sandboxing for secure execution. And each component is separately deployable and scalable.
The user agent is usually a web or mobile app like coderoma.com. It displays the problem description and provides the user with a code editor to write and submit code.
When the user submits the code, the client will get a token since it is an async call. Then the client polls the server for the submission status.
Please see Public API Choices for the protocols we can choose from. And let’s design the interface itself here and GraphQL for example:
type Query {
problems(id: String): [Problem]
languageSetup(id: String!, languageId: LanguageId!): LanguageSetup
submission(token: String!) Submission
}
type Mutation {
createSubmission(
problemId: String!
code: String!
languageId: LanguageId!
): CreatedSubmission!
}
enum LanguageId {
JAVA
JS
ELIXIR
# ...
}
type Problem {
id: String!
title: String!
description: String!
supportedLanguages: [Float!]!
}
type LanguageSetup {
languageId: LanguageId!
template: String!
solutions: [String!]!
}
type Status {
id: Float!
description: String!
}
type Submission {
compileOutput: String
memory: Float
message: String
status: Status
stderr: String
stdout: String
time: String
token: String
}
type CreatedSubmission {
token: String!
}
The API layer records the submission in the database, publishes it into the queue, and returns a token for the client’s future reference.
Code execution engine (CEE) polls the queue for the code, uses a sandbox to compile and run the code and parses the metadata from the compilation and execution.
The sandbox could be LXC containers, Docker, virtual machines, etc. We can choose Docker for its ease of deployment.
I am recently learning Elixir and creating an online judge coderoma.com for my daily practice. It now supports Elixir and JavaScript. And I am adding more languages (like Java) and problems to it.
We may host future events to improve your coding skills. Join us at https://t.me/coderoma for the English community or use your WeChat to scan the following QR onetptp
and reply 刷题
for the Chinese community.
The main features of PWA are as follows.
PWA itself emphasizes Progressive (Progressive) in two perspectives.
Google’s “Progressive Web App Checklist” defines those minimum requirements for PWA.
A PWA combines the benefits of both the Web App and Native App and gives us the following features.
More specifically, what is the advantage of PWA over the native app? Openness and index-ability. Users can hardly install a native app instantly and search across native apps seamlessly.
The table below shows the comparison between t raditional Web App, Native App, and PWA for each feature.
Installable | Linkable | User experience | User stickiness | |
---|---|---|---|---|
Traditional Web | ❌ | ✅ | ❌ | ❌ |
Native App | ✅ | 😐 | ✅️ | ✅ |
PWA | ✅ | ✅ | ✅ | ✅ |
User experience. Back to 2015, frontend developers spend a lot of time optimizing the web by speeding up the rendering of the initial page, making the animation smoother, etc. However, the native app still wins regarding the user experience.
User retention. Native apps can be pinned onto the mobile phone’s home screen and bring the users back into the app by notifications, while the web apps cannot.
Leveraging device APIs. Android and iOS provide abundant device APIs that native apps can easily use with the user’s permission. However, back then, the browser does not fully support them.
Google’s tutorial of Why Build Progressive Web Apps summarizes the problem as “Broad Reach, Low Engagement”.
To tackle the disadvantages of web apps in the mobile age, PWA comes into being.
]]>The plant-based diet has been proven to have a much positive result on health than other foods we have in the US society. A China-Cornell-Oxford research project studied the menu of Chinese people in the 1980s. They found the Guizhou Province has the lowest mortality rate of coronary disease among men under 65, where people consumed the least animal-based food.
Additionally, plant-based diets can facilitate the patient’s recovery. Lifestyle-medical pioneers Nathan Pritikin and Dean Ornish put patients with advanced heart disease on plant-based diets, and then they witnessed significant improvements in alleviating their symptoms. For example, the harmful plaque in patients’ arteries dissolved faster than usual.
Four servings of fruit, including one type of berries, per day, is essential in a healthy diet. A single extra serving of fruit each day has proven to result in a 24-percent decline in the possibility of having Chronic Obstructive Pulmonary Disease. Notably, berries are particularly crucial due to their positive influence on liver function, cancer-resistance, and immune system. In 2014, a study taken among 14 patients revealed that their polyp load declined remarkably after nine months of having black raspberries.
Vegetables play a vital role in preventing diseases. Known as “queen of greens”, kale can reduce people’s cholesterol levels. In a three-month study taken in 2008, high-cholesterol-level patients were asked to take three to four shots of kale juice every day. The proportion of beneficial cholesterol increased significantly, which was equal to the effect of running for 300 miles. Also, cruciferous vegetables help boost liver and lung functions.
Thus it is advised that two of the five daily servings should be leafy veggies (e.g., kale, arugula, and chard). Another two could be carrots, beets, or mushrooms. And finally, one serving of cruciferous vegetables like broccoli, cabbage, or cauliflower.
It is recommended by the American Institute for Cancer Research that beans or legumes should be included in every meal since they contain an animal-free protein as well as fibers. Navy and pinto beans can also be a good alternative to decrease bad cholesterol for people who don’t like soy much. They can also function in slowing down sugar absorption and relaxing stomach. Similar to beans, people should also take whole grains three times daily. In 2015, research found that people who involve whole grains in diets live longer.
The Global Burden of Disease Study conducted between 1990 and 2010 found that eating too few seeds and nuts were the third-leading dietary cause for death and disability all over the world. Even a single serving of brazil nuts is equivalent to statins medicine in lowering cholesterol levels. Nuts and seeds facilitate the detoxification of excess iron. They can also build up bone density. Among all the seeds, chia, hemp, pumpkin, sesame, and sunflower seeds are recommended. It is an excellent alternative to sauces and dressings of your daily meals.
Not only can herbs and spices add flavor to a dish, but they are also essential in preventing diseases, especially cancer. Among all the food groups, they have the highest levels of antioxidants. In a study taken in 2010, Alzheimer participants who took saffron received better cognitive function outcomes than those who had a placebo. Additionally, spices like cloves and cinnamon can reduce depression because of an enzyme inhibited in them.
It is worth mentioning that turmeric is proven to be the best in preventing cancers among all the herbs and spices. Due to the fact that turmeric disappears rapidly, having it with black pepper can slow down the process and help absorption. Curry powder is a decent choice since it usually contains both pepper and turmeric. However, turmeric isn’t for everybody. People with gallstones and kidney stones should restrict their intake.
Theoretically, a human should drink five 12-ounce beverages every day, and pure water is always the best choice. It is claimed by many articles that drinking eight glasses of water per day is a must, yet there is barely scientific evidence on that. Our daily intake of water comes from not only beverages but also fruits and vegetables.
In addition to water, coffee and tea can also be a good alternative since they are all good for health to some extent. For instance, research conducted by Tufts University revealed that tea functioned significantly in dropping blood pressure.
]]>Are Square and Stripe acquirers? No, they are payment aggregators.
Authorization: The cardholder provides the card / card info to the merchant, who processes it and sends (card number, amount, merchant ID) to the acquirer. The acquirer contacts the issuer via the electronic payments networks. The issuer checks against the credit line and fraud-prevention system, and then authorize or decline the transaction.
Clearing / Settlement
Too often, we want to explain an idea thoroughly, but revealing too much detail is not good for people to understand. They will soon forget all the details and even the core message. We should simplify our message and understandably convey the idea, just like journalists would create good headlines for their reports to grasp readers’ attention.
The human brain would neglect selectively to things familiar to save energy. Only something surprising can draw its attention. The fact implies sticky ideas are also unexpected. It proves to be effective to use curiosity gaps to grasp attention. If you present some surprising facts or statistics in your idea, curiosity will drive people to get more information.
Abstract terms are hard to understand, let alone to be remembered. When communicating an idea, we would better use concrete and understandable terms, along with examples and descriptive imagery.
Ideas ought to be credible if they want to be spread out. Generally, there are three ways to add credibility to a plan. The story has experts or people with relevant experiences to back up Use realistic facts and statistics to add credibility to the story Encourage the audience itself to be a reference
Imagine we have the campaign to ask people to donate to starving African children. There are two options: presenting the population of starving children, or showing a picture of a child in need of a donation. Comparatively speaking, people are more likely to take action upon the latter because it appeals directly to human emotions. Therefore sticky ideas should focus on emotional triggers instead of dry facts.
It is a common mistake to focus on an empty slogan without any story when people are communicating ideas. A slogan is sticky, but it can not inspire people to take action like a story. For example, Subway has benefited immensely from the true story of Jared Fogle, who was an overweight man but managed to slim down by having two Subway meals per day.
]]>To create businesses that last for generations, we should get back to long-term thinking and focus on making products people want instead of striving for short-term revenue. Simon Sinek, the author of The Infinite Game, introduces how to equip your business with the infinite mindset.
As Adam Smith put in the book The Wealth of Nations, the interests of consumers should come before the interests of the company. However, in 1970, Milton Friedman published an article writing that the primary responsibility of any free-market enterprise is to make money for shareholders, which signals a shift from being consumer-centric to being shareholder-centric.
A Just Cause is an inspirational goal that encourages your employees to fight for. If companies strive for longevity in the Infinite Game, their goals need to be consumer-centric. If they do not follow this principle, take the GPS device company Garmin, for example. It claimed to be “the global leader in every market we serve” and mentions nothing about their customers in the vision. Then no wonder why it is only worth one-third of its value in 2007.
A company culture of distrust is fatal to business operations. If employees do not trust the company, poor performance, or even unethical behaviors may take place. And this is due to the simple reason that employees don’t know whether to speak up honestly when something unpleasant happens.
Such is the case of Ford Motor Company before the year 2006. The CEO at that time had a habit of blaming and even firing those who brought bad news to him. And then, gradually, employees only reported good news on meetings. The culture of distrust was turned around only after Alan Mulally became the new CEO, who took actions to encourage everyone to bring up bad news.
Whether in the sports field or the business world, a good opponent forces you to improve yourself and learn new techniques. When Allan Mulally became the CEO of Ford Motor Company, Ford had lost 25 percent market share over the past 15 years. Instead of adopting promotion strategies, Allan turned to study rivals like Toyota and Lexus, trying to figure out why consumers preferred those cars over Ford.
It is the same case with Steve Jobs. He changed Apple’s plans instantly when seeing Xerox working on the GUI technology and decided to implement this new technology on Apple’s new computers. Now you can see how successful that move is. And this cannot happen if Jobs cannot embrace new technologies with his flexibility and fast execution.
]]>Join us on Telegram or WeChat(id: onetptp) to discuss and upgrade your PM skills.
Listen to customers
Fast execution
An excellent copywriting creates emotional resonance with customers. And emotions push people to act. If your copywriting hits the bottom of the consumers’ hearts, customers will spend much time and money on your products.
More than rhetoric, creative writing deals with real problems. Using real-world practices to improve your creativity:
Telling a secret is a beautiful way to ignite customers’ interest – Most people are curious about things behind-the-scenes. Doing this improves the click-through rate (CTR).
Straight compliments satisfy customers’ self-esteem. People like to feel unique. This method originates from the three types of persuasion by Aristotle.
It does not mean you can completely ignore the grammar. Our ultimate goal is to convey the message to customers. Sometimes, concise and crisp expressions are the most important, instead of grammar.
]]>When Julie Zhuo just became a manager of Facebook’s design team, she thought her job was to hold meetings with team members to follow up on their progress at work and provide feedback. It takes almost ten years for her to realize that a manager should strive for improved outcomes from the team and focus on broader issues such as how to help her team to work more effectively together instead of daily activities.
Andy Grove, in his book High Output Management, believes that a manager’s output = The output of his organization + The output of the neighboring organizations under his influence.
Providing feedback to team members is indispensable to the job of a manager. But sometimes your criticism might hurt someone’s feelings. Julie suggests two solutions for this situation. One way is to keep your feedback activity-specific. You can provide feedback specific to a task just completed by email or face-to-face communication. This way guarantees the criticism is only for the work someone has done, not the person. The other way is to bring in multiple perspectives. Besides your own opinion, you can share with the team what others think.
Meetings are usually considered as bureaucratic and a waste of time, especially when being held without a purpose. An agenda is not enough to make a productive meeting. In addition, the meeting organizer needs to have a clear idea of the outcome. The outcome could be to present and share information or to make a decision. In the latter situation, everyone who is going to be impacted directly by the decision should attend the meeting. The manager should ensure all the relevant information is presented objectively, and everyone’s voices are heard and equally respected.
Hiring should not be taken as finding someone to fill a vacancy as soon as possible regardless of the skills and experiences. It should be planned in advance. At the beginning of each calendar year, Julie creates a list of vacancies that needs to be recruited for. She suggests a few questions for managers to consider before tailoring their own recruit plans.
Jocko Willink, one of the authors, held all the responsibility for an accident where a soldier lost his life in friendly fire. By doing this, he managed to keep his job because his superiors knew good leaders take responsibility for mistakes and actively look for ways to improve. If the leader makes an excuse to pass the buck, his subordinates will then do the same.
On the battlefield, when Willink was told that his elite team would be fighting side-by-side with the newly created Iraqi army, he doubted the capacity and loyalty of the Iraqi army as well as the correctness of the command. But later, he gradually realized this action could help the US forces to withdraw from Iraq. Then Willink passed his conviction onto his team, and then they finally accomplished the mission successfully.
Leaders should fully understand the importance of every mission and make sure every member is on the same page before carrying it out. If you consider the order received as questionable, think twice before speaking out against the plan. You may also try to seek explanations from your superiors.
“Cover and move” is one of the most fundamental Navy SEAL tactics, which indicates sometimes you need to cooperate with your allies. Leif Babin, the other author, failed to employ this tactic and put his team in extreme danger, which could have been avoided. Leaders should keep an eye on other teams that could provide strategic support instead of competing with them.
In Ramadi, Babin’s team was deep in enemy territory without backup. One team member was wounded and exposed. There was a bomb at the exit. Attention was required for a few problems at the same time. Babin calmly assessed the scenario, sorted out the top three priorities, and managed to escape from the dangerous situation.
In the battlefield where complicated situations often occur, leaders have to stay calm and find the optimal solution. That’s why “prioritize and execute” is thought as a useful principle. It is essential for leaders to decide on the top priority and then focus on it. After the problem is solved, you can move to the next priority and take action.
Before an operation to rescue an Iraqi hostage, Babin fully considered the potential target around the hostage, including explosives and guns, and moved forward as planned, mitigating all the risks.
Creating a comprehensive plan helps to identify and mitigate risks in advance and improve the possibility of success. Besides, leaders should keep members informed of these contingency plans. Concentrate on the risks that can be controlled and be aware that there are always some risks that can not be mitigated.
]]>The composition of your tech knowledge intake
Perception of Value-frequency
In addition to disclosing the unknown unknown, the community discussions creating values in accessibility, building connections, and novelty.
Group Discussion | Published Contents | |
---|---|---|
accessibility | low | high |
building connections | more | less |
novelty | high | low |
Wechaty: a Bot SDK for Wechat Individual Account
Two-layer Biz Structure:
To Business | Social CRM, AI-powered Chatbot |
---|---|
To Developer | Bot SDK and Service Token |
Other examples:
Discussion groups for specific topics posted in a popular online forum could attract hundreds of people once.
e.g., Blockchain Random Discussion Group
What kinds of discussions happen in the group?
CSCH is a community ranging from learners to experienced software engineers who come together to discuss programming, interviewing, career advancement, and, ultimately, how to be better engineers.
What kinds of discussions happen in the group?
Twitter + Engineering Blog + Discussion Groups
Nowadays, many of the most prominent companies in the world, such as Facebook, possess a significant amount of intangible assets, including software, branding, and development capacity, instead of real estate or factories. Companies built with intangible assets, for some reason, behave differently from those reliant on physical assets. The book Capitalism without Capital introduces three main characteristics of intangible assets.
Businesses based on physical assets have one disadvantage in common: they are easy to be limited. When we need more production capacity, we have to invest more tangible assets. However, intangible assets do not have such limits, and they can expand rapidly in no time, which is particularly evident in the tech industry. For example, one mobile app can attract millions of downloads.
Intangible-intensive companies can likely grow incredibly large and finally become monopolies, creating enormous challenges to new entrants in the same industry.
Physical assets are relatively more stable than intangible assets in terms of value. Although physical assets may depreciate, they can always find a buyer in the market at a lower price. The case of intangible assets is a bit complicated. It’s hard to calculate the investments in intangible assets and recover the costs if anything goes wrong.
There is no mature secondary market for brands: on the one hand, the brand value is difficult to estimate; on the other hand, the brand may lose all the value if the business fails. Therefore, intangible assets are high-risk investments and may drop to zero overnight.
Your competitors can steal your intangible assets with little difficulties. When a tech company solves a problem innovatively, many imitators and competitors spring up. iPhone is a good example. And in order to obtain a larger market share, competitors always keep improving the current technology and try to innovate further.
However, being easy to be duplicated brings up a problem of abuse. Policymakers in the intangible economy should enhance the protection of intellectual property rights.
]]>At the beginning of the book The Culture Code, the author introduces a funny competition held among groups of kindergartners, business school students, and lawyers in which the participants need to create the tallest possible structure with uncooked spaghetti, tape, strings, and a marshmallow. Unexpectedly, the game ends with the triumph of kindergartners. How can it happen? When looking back, we discover that business school students usually analyze the problem first, discuss the right strategy, and quietly form a hierarchy. At the same time, kindergartners just start experimenting together and keep trying.
From the different approaches, we can see a good group culture, which can boost the overall performance values, more internal interaction and communication than the skills of group members.
There are three skills to create this kind of group culture.
People can perform at their best in a familiar environment, and that’s why creating a safe working environment is so crucial. The sense of safety usually comes from internal familiarities and connections. If you want to make others feel relaxed and safe, it’s essential to let them know you are paying attention to what they have to say. Sometimes, proper feedback is needed too, which can both increase interactivity and let people feel they are needed.
Although it might sound strange, showing your vulnerabilities actually helps to improve the group performance. We always look at the ways people around us behave and pick up some patterns. Admitting your shortcomings to others indicates they can do the same too. And this will enhance the mutual trust within the group.
Meanwhile, sharing vulnerabilities also conveys the expectation of cooperation. When group members know you rely on their help, they can feel comfortable to rely on you in return. Then everyone is going to know he or she does not have to handle everything on their own.
The pursuit of a common goal is critical to group performance. The common goal refers to beliefs and values behind people’s actions. Gabriele Oettingen, a psychology professor at New York University, has proved in several studies that, communications over the common purpose can help to unite members and achieve goals.
Repetition is necessary for emphasizing the common purpose within the group. You can put it over again and again in regular meetings or make it into short tag lines. Repeat ten times or a hundred times if necessary.
]]>Analogy: it’s aggression, as the Allied invasion of Normandy on D-Day, Our long-term goal is to take over the mainstream market that is currently dominated by an entrenched competitor.
Solution: focus on a niche market that is
If we do not take the niche, we do not worry about our next targets.
The niche-and-next strategy is counterintuitive and thus hard to stick to. If we do not adhere to it, it is like lighting a fire without kindling.
It is fatal to be a sales-driven company; our company should be a market-driven one. Unfortunately, following this strategy takes discipline because leaders can hardly resist the temptation to make short-term money.
The sole goal of the company at this stage should be creating a pragmatist customer base that is reference-able for the mainstream markets.
To achieve the goal, we must ensure the first set of customers completely satisfy their buying objectives with the whole product — a generic product that is needed for the customer to have a compelling reason to buy. The key indicator of this effort is the word-of-mouth reputation among buyers.
Another reason to be niche focused is that we need to achieve market leadership because pragmatists customers want to buy from market leaders. However, you are small and are still crossing the chasm, so the only available strategy is to take a “big fish, small pond” approach.
]]>That is because most people don’t want to spend much time learning how things work. Instead, they prefer to try and figure things out on their own simply by clicking around.
From the users’ perspective, a good website allows them to find what they need by playing around. Based on the book Don’t Make Me Think, Revisited, we have summarized four guidelines for creating a website of good usability.
When a user lands on a website for the first time, he can hardly estimate the website’s scale. He will probably choose to opt-out for not knowing how the website is organized. That’s why a navigation bar is so necessary. On each page, the navigation should enable users to locate where they are, how they can get back to the homepage, where they can search for keywords and other extra information about how to use the website. Most of all, navigation must be simple and straightforward; otherwise, users may get confused and just click away.
The home page is likely to be the most frequently visited page of a website, and its importance is self-evident. Users’ first impression of a website is determined by their first impression of the home page. An impressive home page is a must.
Besides, we also need to make sure that we have delivered the most important message to visitors, which is the goal of our website. The book suggests an effective way to communicate with readers on the home page would be placing a tagline describing our mission next to the website logo.
When we visit a website, we don’t read the text line by line. Instead, we often scan the text to retrieve the information we need. If you want to convey a specific message to visitors, you should learn to make use of visual hierarchies. The rule is simple. Key messages should be highlighted. So users can understand where to focus on and click. Hiding important information will only annoy visitors.
A few years ago, being responsive was only a plus, but nowadays, it has become a must.
Problems with Comic Sans:
Dating back to the early eighteenth century, most of the ads were just informational instead of being persuasive. This situation did not change until Benjamin Day launched his own newspaper in 1833. In order to obtain a large audience, he set the price at a penny while rivals sold at six cents per copy. The loss was inevitable. However, he began to invite businesses to publish ads in his newspaper with an exposure fee. As a result, the newspaper became phenomenal. Because of Benjamin Day, the world started to realize the value of selling the attention of the audience.
Advertising became more methodical at the beginning of the 20th century, and it was considered as a science. People start to use advertisements specifically for grabing attention. Demand engineering was one of these scientific approaches. It advertised a problem that was never recognized, or sometimes totally fabricated at first, and then claimed the solution of using a certain product. Advertisers of the 1920s also realized that a good reputation could be engineered as well by creating and publicizing a brand.
The main focus of advertising was in public space before the 1920s. Things changed afterward. Radio advertising became very popular, and advertisers were willing to sponsor the radio content, and hear their names being mentioned during the broadcast, which could draw immediate attention from a large audience. However, the role that radio played in advertising was soon replaced by television, which turned out to be a much more effective way of attracting attention till now.
The emergence of the Internet brought another round of ad revolution, during which emails showed up as a new form of advertising. A survey in 1973 found emails comprised 75 percent of all network traffic. Email is very effective because it rewards people. Receiving an email makes people feel good. Search engines such as Google also become significant sources to harvest lots of attention from users. In order to monetize the attention, the Googlers employed an advertising tool called Adwords to display relevant ads to users based on what they are searching for, along with the results.
]]>Why are we so easily distracted? There might be external causes sometimes. However, in most cases, it turns out distraction has internal sources. We are distracted because we want to escape from discomfort. To avoid distraction, we need to solve the problem from inside.
Next time you feel inclined to distraction, try to record your feelings and what triggers that. That’s how you can identify internal triggers in the first place. Then you can try to avoid the triggers by making tasks more fun.
To have a plan can prevent you from distraction because you will know what exactly you are striving for. However, scheduling for work is not the best place to start. On the contrary, plan for yourself and your relationships first, and then you will not escape to your hobbies in the middle of work.
Office distractions such as email notifications are typical external triggers. Let others know that you need to be entirely focused on the task at hand, so they are not supposed to interrupt you. Also, learn to sort your emails more effectively and make sure only a few emails demand your attention every day. Other than emails, there are other forms of distraction in the workplace. Learn to organize them in the least distracting way.
You have to be aware of the fact that the battle between you and distraction is not a one-day fight. Maybe you can try an APP to block your access to distracting. Or find a study buddy to focus together. Imposing fines for missing targets also sounds practical, which has been testified by the author.
Dysfunctional work culture is the beginning of endless distraction, in which employees are overburdened and even required to answer emails after work. Employers should create a platform that enables employees to give feedbacks safely without be worried about getting fired. Step by step, the company can head towards a functional work culture.
]]>Data is vital to business. Entrepreneurs need data to convince others that their ideas will work. Sometimes, entrepreneurs tend to overestimate their success but data will not lie. Data helps founders to stay grounded in reality. However, personal judgement of what data to pursue is also important. Don’t be just a slave to numbers.
In order to stay data-informed, you need to find some metrics which can provide meaningful data. Good metrics have three characteristics:
The Lean Analytics framework suggests a start-up will go through five stages:
To achieve success, founders must focus on one metric that’s most critical. Knowing what is the most important metric prevents you from getting lost in the data world.
There is no best metric in general. In different industries, the best metric differs. For E-commerce companies, the most important metric is revenue per customer. However, for media sites, the best metric is the click-through rates.
]]>Charles Handy makes an analogy as his road to Davy’s Bar. Turn right and go up the hill when there is half a mile to the Davy’s Bar. However, when he realized he was on the wrong way, he arrived at Davy’s Bar already.
The growth curve is usually in an “S” shape, and we call it S-curve or sigmoid curve. To keep the overall growth rate high, you have to develop your second S-curve before it is too late to invest your time and resources.
Intel’s CPU, Netflix’s video streaming, Nintendo’s gaming, Microsoft’s cloud are all excellent examples of the second-curve-driving businesses.
How to find and catch the second curve takes vision and execution. You have to input more information and continuously sort them to identify the best opportunities. And then, once a chance identified, you need a reliable team to fight the battle and figure out whether it really works.
What makes you succeed may not make you succeed again. There is always a limit to growth. The second curve theory helps us reflect on why and how to embrace the change and live a more thriving life.
]]>Internet-scale web services deal with high-volume traffic from the whole world. However, one server could only serve a limited amount of requests at the same time. Consequently, there is usually a server farm or a large cluster of servers to undertake the traffic altogether. Here comes the question: how to route them so that each host could evenly receive and process the request?
Since there are many hops and layers of load balancers from the user to the server, specifically speaking, this time our design requirements are
Note: If Service A depends on (or consumes) Service B, then A is downstream service of B, and B is upstream service of A.
Why is it hard to balance loads? The answer is that it is hard to collect accurate load distribution stats and act accordingly.
Random and round-robin distribute the traffic by requests. However, the actual load is not per request - some are heavy in CPU or thread utilization, while some are lightweight.
To be more accurate on the load, load balancers have to maintain local states of observed active request number, connection number, or request process latencies for each backend server. And based on them, we can use distribution algorithms like Least-connections, least-time, and Random N choices:
Least-connections: a request is passed to the server with the least number of active connections.
latency-based (least-time): a request is passed to the server with the least average response time and least number of active connections, taking into account weights of servers.
However, these two algorithms work well only with only one load balancer. If there are multiple ones, there might have herd effect. That is to say; all the load balancers notice that one service is momentarily faster, and then all send requests to that service.
Random N choices (where N=2 in most cases / a.k.a Power of Two Choices): pick two at random and chose the better option of the two, avoiding the worse choice.
Local LB is unaware of global downstream and upstream states, including
There are three options to collect load the stats accurately and then act accordingly:
Dropbox Bandaid team chose the third option because it fits into their existing random N choices approach well.
However, instead of using local states, like the original random N choices do, they use real-time global information from the backend servers via the response headers.
Server utilization: Backend servers are configured with a max capacity and count the on-going requests, and then they have utilization percentage calculated ranging from 0.0 to 1.0.
There are two problems to consider:
core.async
.Multi-layer architecture
The abstraction of this problem is to find documents by prefixes and terms in a very large number of elements. The solution leverages these four major data structures:
InvertedIndex<prefixes or terms, documents>
: given any prefix, find all the document ids that contain the prefix.for each document, prepare a BloomFilter<prefixes or terms>
: with user typing more, we can quickly filter out documents that do not contain the latest prefixes or terms, by check with their bloom filters.ForwardIndex<documents, prefixes or terms>
: previous bloom filter may return false positives, and now we query the actual documents to reject them.scorer(document):relevance
: Each partition return all of its true hits and scores. And then we aggregate and rank.In details, Lyft’s advertisements should meet requirements as below:
However, the biggest challenge is to manage all the processes of cross-region marketing at scale, which include choosing bids, budgets, creatives, incentives, and audiences, running A/B tests, and so on. You can see what occupies a day in the life of a digital marketer:
We can find out that execution occupies most of the time while analysis, thought as more important, takes much less time. A scaling strategy will enable marketers to concentrate on analysis and decision-making process instead of operational activities.
To reduce costs and improve experimental efficiency, we need to
The marketing performance data flows into the reinforcement-learning system of Lyft: Amundsen
The problems that need to be automated include:
The tech stack includes - Apache Hive, Presto, ML platform, Airflow, 3rd-party APIs, UI.
The lifetime value of a user is an important criterion to measure the efficiency of acquisition channels. The budget is determined together by LTV and the price we are willing to pay in that region.
Our knowledge of a new user is limited. The historical data can help us to predict more accurately as the user interacts with our services.
Initial eigenvalue:
The forecast improves as the historical data of interactivity accumulates:
After LTV is predicted, the next is to estimate budgets based on the price. A curve of the form LTV = a * (spend)^b
is fit to the data. A degree of randomness will be injected into the cost-curve creation process in order to converge a global optimum.
Bidders are made up of two parts - the tuners and actors. The tuners decide exact channel-specific parameters based on the price. The actors communicate the actual bid to different channels.
Some popular bidding strategies, applied in different channels, are listed as below:
We have to value human experiences in the automation process; otherwise, the quality of the models may be “garbage in, garbage out”. Once saved from laboring tasks, marketers can focus more on understanding users, channels, and the messages they want to convey to audiences, and thus obtain better ad impacts. That’s how Lyft can achieve a higher ROI with less time and efforts.
]]>Suppliers provide their room details in the inventory. And users can search, get, and reserve rooms accordingly. After reserving the room, the user’s payment will change the status
of the reserved_room
as well. You could check the data model in this post.
room_id
, check all occupied_room
today or later, transform the data structure to an array of occupation by days, and finally find available slots in the array. This process might be time-consuming, so we can build the availability index.room_id
, always create an entry for an occupied day. Then it will be easier to query unavailable slots by dates.If it is a hotel booking system, then it will probably publish to Booking Channels like GDS, Aggregators, and Wholesalers.
To sync data across those places. We can
Data model: double-entry bookkeeping
To execute the payment, since we are calling the external payment gateway, like bank or Stripe, Braintree, etc. It is crucial to keep data in-sync across different places. We need to sync data across the transaction table and external banks and vendors.
The notification system is essentially a delayer scheduler (priority queue + subscriber) plus API integrations.
For example, a daily cronjob will query the database for notifications to be sent out today and put them into the priority queue by date. The subscriber will get the earliest ones from the priority queue and send out if reaching the expected timestamp. Otherwise, put the task back to the queue and sleep to make the CPU idle for other work, which can be interrupted if there are new alerts added for today.
]]>Big Picture: Client-server
The Key-value server consists of a fixed-size hash table + single-threaded handler + coarse locking
How to handle collisions? Mostly three ways to resolve:
See Data Partition and Routing
See Key value cache
empathy / perspective-taking is the most important.
choose a sustainable architecture to reduce human resources costs per feature.
adopt patterns and best practices.
avoid anti-patterns
effective refactoring
Everyone has advice about how to manage money. Search Google for “manage money,” and you’ll get back over 1,690,000,000 links. You’ll find tons of life-hacking or self-help articles and books. You’ll find professional coaches or courses witch will coach you for a fee. You’ll find financial and investment services. Feel free to try what appeals to you and grow your assets through trials and errors.
I think the most important thing to remember is that asking the question of money comes from fear and self-doubt. We all fear change. We all doubt our ability to make more money.
Instead of spending time worrying and doubting, focusing the opposite — your confidence. If you are playing poker and with few chips, you can only make small bets and only win a small amount of money. When you have a lot of chips, you can make big bets and win big. You have more room for taking risks. You can try things which you cannot try when you have fewer chips.
Here is the magic - by understanding more of your financial status, you gain confidence! With more confidence, we can make better judgment and would like to bet the best amount for more significant success, and then win more.
Know your expenses and plan for next spending>
Where is the end of the wining-more concern? People often talk about the buzzword of financial freedom. However, talk is cheap, but bookkeeping precisely answers the question.
Unfortunately, bookkeeping is not easy in our modern life. We are in a new age of abundance. We have a lot of accounts - cash, bank accounts, payment apps, credit cards, stock or crypto broker accounts, discount cards, … We have assets like houses, cars, gold, jewelry, … To make things even worse, some of us may live across countries and have to deal with different currencies. How could we draw an accurate map of our financial life and navigate through the future uncertainties?
By “accurate map of our financial life,” I mean these four primary financial statements:
With beancount.io, you can quickly generate statements like the above. But wait… How to prepare data for these statements?
To ensure the accuracy and internalize the error detection into the system, double-entry bookkeeping requires every entry to an account has at-least a corresponding entry to a different account. One transaction involves at least two accounts with two operations - debit (+) and credit (-).
1970-01-01 open Income:BeancountCorp
1970-01-01 open Assets:Cash
1970-01-01 open Expenses:Food
1970-01-01 open Assets:Receivables:Alice
1970-01-01 open Assets:Receivables:Bob
1970-01-01 open Assets:Receivables:Charlie
1970-01-01 open Liabilities:CreditCard
2019-05-31 * "BeancountCorp" "Salary of May 15th to May 31st"
Income:BeancountCorp -888 USD
Assets:Cash 888 USD
2019-07-12 * "Popeyes chicken sandwiches" "dinner with Alice, Bob, and Charlie"
Expenses:Food 20 USD
Assets:Receivables:Alice 20 USD
Assets:Receivables:Bob 20 USD
Assets:Receivables:Charlie 20 USD
Liabilities:CreditCard -80 USD
As you can see in the two examples above, every transaction must fulfill the accounting equation.
Assets = Liabilities + Equity(aka Net Assets)
We used the Beancount syntax by Martin Blais and the web project Fava by Jakob Schnitzer to build this website. And it will alert you if any transaction has any legs not summing to zero.
Now you understand how we enforce the correctness of the ledger. But you may ask what are those “accounts”?
Thinking your assets as water running in and out of different buckets and “accounts” are those buckets holding your money. With double-entry bookkeeping, it becomes obvious how money is flowing across different accounts, just like how water is flowing across different buckets.
Beancount.io introduces five kinds of accounts.
Equity = Assets - Liabilities
and it reflects how wealthy you are.Now you can open your customized accounts with those keywords above:
1970-01-01 open Assets:Cash
1970-01-01 open Assets:Stock:Robinhood
1970-01-01 open Assets:Crypto:Coinbase
1970-01-01 open Expenses:Transportation:Taxi
1970-01-01 open Equity:OpeningBalance
Yes, you can track your investment with beancount.io. For example, we buy 10 Bitcoins at the price of $100 in 2014:
2014-08-08 * "Buy 10 Bitcoin"
Assets:Trade:Cash -1000.00 USD
Assets:Trade:Positions 10 BTC {100.00 USD}
And then three years later, you sell them (originally with costs of $100 per unit annotated with {100.00 USD}
) at the price of $10,000 per unit annotated with @ 10,000.00 USD
.
2017-12-12 * "Sell 2 Bitcoin"
Assets:Trade:Positions -2 BTC {100.00 USD} @ 10,000.00 USD
Assets:Trade:Cash 20,000.00 USD
Income:Trade:PnL -19,800.00 USD
Or the same transaction with @@ 20,000.00 USD
means that at the price of $20,000 in total.
2017-12-12 * "Sell 2 Bitcoin"
Assets:Trade:Positions -2 BTC {100.00 USD} @@ 20,000.00 USD
Assets:Trade:Cash 20,000.00 USD
Income:Trade:PnL -19,800.00 USD
The sum of all legs of the transaction, including -2 BTC {100.00 USD}
, are still, as always, zero.
The costs {100.00 USD}
tag is important because you might have bought the same commodity at different costs.
100 BTC {10.00 USD, 2012-08-08}
10 BTC {100.00 USD, 2014-08-08}
If you want to simplify the process, you can set up the account at the beginning with FIFO or LIFO. FIFO stands for first in, first out, while LIFO stands for last in, first out. In the US, IRS uses FIFO to calculate your PnL and tax accordingly.
1970-01-01 open Assets:Trade:Positions "FIFO"
And then when you sell it in shorthand like -2 BTC {}
, beancount will apply FIFO strategy automatically and sell the oldest commodity.
Beancount.io is such a cloud service for recording your financial transactions in text files, visualize them into financial statements (income statement, balance sheet, trial balance, etc.), and helps you live a better financial life. Sign up now - It’s in Promotional Period and Free!
]]>Log v.s Metric: A log is an event that happened, and a metric is a measurement of the health of a system.
We are assuming that this system’s purpose is to serve metrics - namely, counters, conversion rate, timers, etc. for monitoring the system performance and health. If the conversion rate drops drastically, the system should alert the on-call.
Two ways to build the system:
The pull model is more scalable because it decreases the number of requests going into the metrics databases - there is no hot path and concurrency issue.
Take a four-step sign up on the mobile app for example
INPUT_PHONE_NUMBER -> VERIFY_SMS_CODE -> INPUT_NAME -> INPUT_PASSWORD
Every step has IMPRESSION
and POST_VERIFICATION
phases. And emit metrics like this:
{
"sign_up_session_id": "uuid",
"step": "VERIFY_SMS_CODE",
"os": "iOS",
"phase": "POST_VERIFICATION",
"status": "SUCCESS",
// ... ts, contexts, ...
}
Consequently, we can query the overall conversion rate of VERIFY_SMS_CODE
step on iOS
like
(counts of step=VERIFY_SMS_CODE, os=iOS, status: SUCCESS, phase: POST_VERIFICATION) / (counts of step=VERIFY_SMS_CODE, os=iOS, phase: IMPRESSION)
Graphana is mature enough for the data visualization work. If you do not want to expose the whole site, you can use Embed Panel with iframe.
]]>Designing a service money transfer backend system like Square Cash (we will call this system Cash App below) or PayPal to
The payment data model is essentially “double-entry bookkeeping”. Every entry to an account requires a corresponding and opposite entry to a different account. Sum of all debit and credit equals to zero.
Transaction: new user Jane Doe deposits $100 from bank to Cash App. This one transaction involves those DB entries:
bookkeeping table (for history)
+ debit, USD, 100, CashAppAccountNumber, txId
- credit, USD, 100, RoutingNumber:AccountNumber, txId
transaction table
txId, timestamp, status(pending/confirmed), [bookkeeping entries], narration
Once the bank confirmed the transaction, update the pending status above and the following balance sheet in one transaction.
balance sheet
CashAppAccountNumber, USD, 100
Similar to the case above, but there is no pending state because we do not need the slow external system to change their state. All changes in bookkeeping table, transaction table, and balance sheet table happen in one transaction.
We solve the i18n problems in 3 dimensions.
accept-language
header.For example, Jane Doe wants to exchange 1 USD with 6.8 CNY with 0.2
bookkeeping table
- credit, USD, 1, CashAppAccountNumber, txId
+ debit, CNY, 6.8, CashAppAccountNumber, txId, @7.55 CNY/USD
+ debit, USD, 0.1, ExpensesOfExchangeAccountNumber, txId
Transaction table, balance sheet, etc. are similar to the transaction discussed in Deposit and Payout. The major difference is that the bank or the vendor provides the exchange service.
poll
: cronjobs (SWF, Airflow, Cadence, etc.) to poll the status for PENDING orders.callback
: provide a callback API for the external vendors.Why is Deduplication a concern?
For the poll
case above, if the external gateway does not support idempotent APIs, in order not to flood with duplicate entries, we must keep record of the order ID or the reference ID the external system gives us with 200, and query GET
by the order ID instead of POST
all the time.
For the callback
case, we can ensure we implement with idempotent APIs, and we mutate pending
to confirmed
anyway.
async design + retry + queuing + time-series DB + security
// POST https://example.com/webhook/
{
"id": 1,
"scheduled_for": "2017-01-31T20:50:02Z",
"event": {
"id": "24934862-d980-46cb-9402-43c81b0cdba6",
"resource": "event",
"type": "charge:created",
"api_version": "2018-03-22",
"created_at": "2017-01-31T20:49:02Z",
"data": {
"code": "66BEOV2A", // or order ID the user need to fulfill
"name": "The Sovereign Individual",
"description": "Mastering the Transition to the Information Age",
"hosted_url": "https://commerce.coinbase.com/charges/66BEOV2A",
"created_at": "2017-01-31T20:49:02Z",
"expires_at": "2017-01-31T21:49:02Z",
"metadata": {},
"pricing_type": "CNY",
"payments": [
// ...
],
"addresses": {
// ...
}
}
}
}
The merchant server should respond with a 200 HTTP status code to acknowledge receipt of a webhook.
If there is no acknowledgment of receipt, we will retry with exponential backoff for up to three days. The maximum retry interval is 1 hour.
x-webhook-signature
SHA256 HMAC signature. Its value is HMAC(webhook secret, raw request payload);
. We generate the secret for the developer to use.Background Knowledge: HMAC (message authentication code). A short piece of information used to authenticate a message — In other words, to confirm that the message came from the stated sender (its authenticity) and has not been changed in transit (its integrity). The integrity can be verified by the shared secret between trusted parties against the digest of the original message.
The webhook gateway service emits statuses into the time-series DB for metrics.
Using Influx DB vs. Prometheus?
I will probably choose InfluxDB for easier maintenance of the monolithic data store.
Depending on how much further data aggregation we need, we can build more advanced data pipeline. However, for just counting success/ failures, a simple time-series DB solves the problem.
]]>Contexts:
PRICE_CHANGE(STOCK_CODE, timeSpan, percentage)
to the notification queue.PRICE_CHANGE
, EARNINGS_CALL
, BREAKING_NEWS
, etc.buy
and sell
orders. Types of orders:
Order(id, stock, side, time, qty, price)
x-axis
with y-axis cast into x-axis
Id Side Time Qty Price Qty Time Side
---+------+-------+-----+-------+-----+-------+------
#3 20.30 200 09:05 SELL
#1 20.30 100 09:01 SELL
#2 20.25 100 09:03 SELL
#5 BUY 09:08 200 20.20
#4 BUY 09:06 100 20.15
#6 BUY 09:09 200 20.15
Order book from Coinbase Pro
The Single Stock-Exchange Simulator
How to implement the price-time FIFO matching algorithm?
How to transmit data of the order book to the client-side in realtime?
How to support different kinds of orders?
SELL or BUY: qty @ price
in the treemap with different creation setup and matching conditions
If you cannot watch the video, here are the words from him.
We went through that stage at Apple where we thought, ‘Oh, we’re going to be a big company, let’s go out and hire professional management.’ We went out and hired a bunch of professional management; it didn’t work at all. Most of them were Bozos. They knew how to manage, but they didn’t know how to DO anything.
]]>If you are a great person, why do you want to work for somebody you cannot learn anything from? And you know what’s interesting - you know what the best managers are? They are the great individual contributors who never ever wanted to be a manager, but decide they have to be a manager because no one else is able to do as good job as them.
Decentralized Identity Foundation builds ecosystem for decentralized identity and ensures interop between all participants.
The problem of decoupling ID from Personally identifiable information (PII). Identity is composed of a deeply personal collection of data that defines us, and your identity should answer to no one but you.
Specifically, challenges are
DIF is the organization uniting the fragmented to solve the DID problem together and build an ecosystem as an industry standard.
Working Groups
Members
ID that is
DID Format: URN
DID methods (further explained below) define how DIDs work with a specific blockchain.
Defining how a DID and DID document are created, resolved, and managed (CRUD) on a specific blockchain.
https://w3c-ccg.github.io/did-method-registry/#the-registry
Examples
on-chain DIDs
did:stack:v0:15gxXgJyT5tM5A4Cbx99nwccynHYsBouzr-3
means the fourth on-chain name was created and initially assigned to the address 15gxXgJyT5tM5A4Cbx99nwccynHYsBouzr
.off-chain DIDs. a.k.a. subdomains
cicero.res_publica.id
is processed by the owner of res_publica.id
but are not owned by it.Demo: https://bitpatron.co/ login with Blockstack
// authRequestJwt
{
"typ": "JWT",
"alg": "ES256K"
}
{
"jti": "4d06f08b-67a7-4f7c-89fc-b8164b81f67a",
"iat": 1563432343,
"exp": 1566110743,
"iss": "did:btc-addr:19sxvnAxPXZYAEdpF7Tti6MSVhxA8PSdCT",
"public_keys": [
"03994ec7b23a8e11e40684c9b2d29febf103bd92c4bbd295f1e2537042c93ac977"
],
"domain_name": "http://localhost:4104",
"manifest_uri": "http://localhost:4104/manifest.json",
"redirect_uri": "http://localhost:4104/",
"version": "1.3.1",
"do_not_include_profile": true,
"supports_hub_url": true,
"scopes": [
"store_write"
]
}
// authResponse
{
"typ": "JWT",
"alg": "ES256K"
}
{
"jti": "30773b78-3595-499f-bbb3-d1e649470c70",
"iat": 1563432894,
"exp": 1566111294,
"iss": "did:btc-addr:1DpKMqxBnuSSQMNun1obciPSfD9rD8KNUH",
"private_key": "redacted - encrypted with transitKey",
"public_keys": [
"027c8547681cc27e27b73ee0f3c0534bdd38993dcb4c1934bf424f0b3a04dcad63"
],
"profile": null,
"username": "kirbystar.id.blockstack",
"core_token": null,
"email": null,
"profile_url": "https://gaia.blockstack.org/hub/1DpKMqxBnuSSQMNun1obciPSfD9rD8KNUH/profile.json",
"hubUrl": "https://hub.blockstack.org",
"blockstackAPIUrl": "https://core.blockstack.org",
"associationToken": "eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzI1NksifQ.eyJjaGlsZFRvQXNzb2NpYXRlIjoiMDNhMTU5YzY4YWQ1ZjFkNzcxMWY2NjJmNThkNjdmMzZlNzY3ZTBjMDBhOTU4ZWY0NzljNzU3MzU0MGFkMzExZjk2IiwiaXNzIjoiMDI3Yzg1NDc2ODFjYzI3ZTI3YjczZWUwZjNjMDUzNGJkZDM4OTkzZGNiNGMxOTM0YmY0MjRmMGIzYTA0ZGNhZDYzIiwiZXhwIjoxNTk0OTY4ODk0LjcwNSwiaWF0IjoxNTYzNDMyODk0LjcwNSwic2FsdCI6IjE4NGVhMWQyMzM3MWQ1MmYyYzhmNTAyOGUwMWYxYmZiIn0.ZceaVcIK2Z8wu6KBYOHQaK7y6BI7NfxrixphOCPs1B4hZcGYDKsuf0anbm4CdAAJbKRifCm-MYHE6fjKD9E7GQ",
"version": "1.3.1"
}
// acctName response
{
"blockchain": "bitcoin",
"status": "submitted_subdomain",
"last_txid": "851ca5e6c06723e61037aa397966aafa1a6dd7159e9e31e53116106b87101886",
"zonefile": "$ORIGIN kirbystar.id.blockstack\n$TTL 3600\n_http._tcp\tIN\tURI\t10\t1\t\"https://gaia.blockstack.org/hub/1DpKMqxBnuSSQMNun1obciPSfD9rD8KNUH/profile.json\"\n\n",
"address": "1DpKMqxBnuSSQMNun1obciPSfD9rD8KNUH",
"zonefile_hash": "a9c016921a9a60e04776251db53a8881e6d128ce"
}
// session
{
"version": "1.0.0",
"userData": {
"username": "kirbystar.id.blockstack",
"profile": {
"@type": "Person",
"@context": "http://schema.org",
"api": {
"gaiaHubConfig": {
"url_prefix": "https://gaia.blockstack.org/hub/"
},
"gaiaHubUrl": "https://hub.blockstack.org"
}
},
"email": null,
"decentralizedID": "did:btc-addr:1DpKMqxBnuSSQMNun1obciPSfD9rD8KNUH",
"identityAddress": "1DpKMqxBnuSSQMNun1obciPSfD9rD8KNUH",
"appPrivateKey": "redacted",
"coreSessionToken": null,
"authResponseToken": "eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzI1NksifQ.eyJqdGkiOiIyNmM4ZGM2Ny1lNzEwLTRlZDUtYmIxYi0yY2I1ODY5YTRkMTEiLCJpYXQiOjE1NjM0MzU2NjEsImV4cCI6MTU2NjExNDA2MSwiaXNzIjoiZGlkOmJ0Yy1hZGRyOjFEcEtNcXhCbnVTU1FNTnVuMW9iY2lQU2ZEOXJEOEtOVUgiLCJwcml2YXRlX2tleSI6IjdiMjI2OTc2MjIzYTIyNjQzMDMzMzMzODMyNjYzNTYxNjIzNjM0Mzg2MTM1MzYzNTM2MzI2MzMzNjEzMTYyMzY2NTYzNjEzNzMzNjU2NjIyMmMyMjY1NzA2ODY1NmQ2NTcyNjE2YzUwNGIyMjNhMjIzMDMyNjEzMzY1NjEzMzY0NjMzNjMyMzgzODY2MzgzMTY2MzAzNDMwMzU2MzY1MzI2MzMzNjUzNjY0MzMzMzMxMzQzNDM0NjY2MTM4MzA2NTY1MzIzMzMyNjEzMjM2NjQzMzMwMzczMzM2MzY2NTMzMzkzNzM3MzczMjMxMzMzNzM5MzcyMjJjMjI2MzY5NzA2ODY1NzI1NDY1Nzg3NDIyM2EyMjM2MzMzOTYzNjMzMDM5MzQzOTYyMzMzNjY0NjI2NTM1MzIzMTM0MzIzMjM3MzA2MzY1NjIzMTM0MzEzNjY2MzA2MzM0NjYzNDM4MzgzMjY0MzUzOTM0MzMzNDM2NjM2NjYyNjEzNzMxNjM2MjM1MzYzMTM2MzczMDY0MzUzMzY0MzQzNTY1MzMzODYyNjIzNzM1NjMzMjMzMzYzNjMyMzUzODMxMzgzOTM3NjYzMjMwNjMzNTM4MzA2NTMyMzEzODM0MzMzNTMwMzMzNjM1NjE2NDM1NjEzMTM4NjY2MjMyNjY2NDM3MzQ2MzY1NjMzNDM0MzI2NTY0MzY2NTY2NjYzMjM5NjIzNjY2Mzk2MTMyMzgzMTM4MzEzMzYyMzMzMDYyMzkzMTYzMzE2MzM1Mzg2MTM4MzgzMTM5MzQ2MjYzMjIyYzIyNmQ2MTYzMjIzYTIyMzkzNzY0MzM2MzM4NjEzOTYxMzQ2NjMyMzkzNjM1MzM2MzM1MzQzNzY1MzIzNTYzMzMzMzM4MzIzMDYxMzczMjYyMzU2NjY0MzQ2NjM1NjE2NDYxNjY2NjMwMzQzNzM1Mzk2NDM0MzI2MTYxMzgzNDM1MzkzNzY2MzY2MjM5NjEyMjJjMjI3NzYxNzM1Mzc0NzI2OTZlNjcyMjNhNzQ3Mjc1NjU3ZCIsInB1YmxpY19rZXlzIjpbIjAyN2M4NTQ3NjgxY2MyN2UyN2I3M2VlMGYzYzA1MzRiZGQzODk5M2RjYjRjMTkzNGJmNDI0ZjBiM2EwNGRjYWQ2MyJdLCJwcm9maWxlIjpudWxsLCJ1c2VybmFtZSI6ImtpcmJ5c3Rhci5pZC5ibG9ja3N0YWNrIiwiY29yZV90b2tlbiI6bnVsbCwiZW1haWwiOm51bGwsInByb2ZpbGVfdXJsIjoiaHR0cHM6Ly9nYWlhLmJsb2Nrc3RhY2sub3JnL2h1Yi8xRHBLTXF4Qm51U1NRTU51bjFvYmNpUFNmRDlyRDhLTlVIL3Byb2ZpbGUuanNvbiIsImh1YlVybCI6Imh0dHBzOi8vaHViLmJsb2Nrc3RhY2sub3JnIiwiYmxvY2tzdGFja0FQSVVybCI6Imh0dHBzOi8vY29yZS5ibG9ja3N0YWNrLm9yZyIsImFzc29jaWF0aW9uVG9rZW4iOiJleUowZVhBaU9pSktWMVFpTENKaGJHY2lPaUpGVXpJMU5rc2lmUS5leUpqYUdsc1pGUnZRWE56YjJOcFlYUmxJam9pTUROaE1UVTVZelk0WVdRMVpqRmtOemN4TVdZMk5qSm1OVGhrTmpkbU16WmxOelkzWlRCak1EQmhPVFU0WldZME56bGpOelUzTXpVME1HRmtNekV4WmprMklpd2lhWE56SWpvaU1ESTNZemcxTkRjMk9ERmpZekkzWlRJM1lqY3paV1V3WmpOak1EVXpOR0prWkRNNE9Ua3paR05pTkdNeE9UTTBZbVkwTWpSbU1HSXpZVEEwWkdOaFpEWXpJaXdpWlhod0lqb3hOVGswT1RjeE5qWXhMak0yT1N3aWFXRjBJam94TlRZek5ETTFOall4TGpNMk9Td2ljMkZzZENJNklqVXhOMkZsWkdVd1ltVmpOMkpqTlRnek5qY3lOREkwT1RsaE1EVm1OVEEwSW4wLjlkY2VHX3I4OVdDSUVsTklseFNtUGxPblhiSVNsZEZDejJxOTJRMnpJSk9XXzhnTjVYT0xsZnNkREJVamlQZlU3eTNyRGFXSUxfTUJicVVnVnBFanhRIiwidmVyc2lvbiI6IjEuMy4xIn0.0Xqtw-71TJ9ybWx4Uxre0Gxkisay20xn1vqwr0WaKvVeCzwv_NO6YZnVOmGPM4cF4wex06yLYWasqQWgCi-m_g",
"hubUrl": "https://hub.blockstack.org",
"gaiaAssociationToken": "eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzI1NksifQ.eyJjaGlsZFRvQXNzb2NpYXRlIjoiMDNhMTU5YzY4YWQ1ZjFkNzcxMWY2NjJmNThkNjdmMzZlNzY3ZTBjMDBhOTU4ZWY0NzljNzU3MzU0MGFkMzExZjk2IiwiaXNzIjoiMDI3Yzg1NDc2ODFjYzI3ZTI3YjczZWUwZjNjMDUzNGJkZDM4OTkzZGNiNGMxOTM0YmY0MjRmMGIzYTA0ZGNhZDYzIiwiZXhwIjoxNTk0OTcxNjYxLjM2OSwiaWF0IjoxNTYzNDM1NjYxLjM2OSwic2FsdCI6IjUxN2FlZGUwYmVjN2JjNTgzNjcyNDI0OTlhMDVmNTA0In0.9dceG_r89WCIElNIlxSmPlOnXbISldFCz2q92Q2zIJOW_8gN5XOLlfsdDBUjiPfU7y3rDaWIL_MBbqUgVpEjxQ"
},
"transitKey": "redacted"
}
Domain Name <--DNS--> IP
Repensented Entity <--Universal Resolver--> Self-sovereign Identifiers
DID <--Universal Resolver--> DID Document
Drivers for Example:
did:stack:
DID registered from BlockStack, like did:stack:v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0
did:btcr
: DID registered from BTCgit clone https://github.com/decentralized-identity/universal-resolver.git
cd universal-resolver/
docker-compose -f docker-compose.yml pull
docker-compose -f docker-compose.yml up
curl -X GET http://localhost:8080/1.0/identifiers/did:stack:v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0 | jq .
{
"redirect": null,
"didDocument": {
"id": "did:stack:v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0",
"service": [
{
"type": "blockstack",
"serviceEndpoint": "https://core.blockstack.org"
}
],
"publicKey": [
{
"id": "did:stack:v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0",
"type": "Secp256k1VerificationKey2018",
"publicKeyHex": "0232131c807c4b184582280bca141f2583f6a1de2e0d3e6984cdb4724527f581fa"
}
],
"@context": "https://w3id.org/did/v0.11"
},
"resolverMetadata": {
"duration": 96,
"driverId": "did-stack",
"driver": "HttpDriver",
"didUrl": {
"didUrlString": "did:stack:v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0",
"did": {
"didString": "did:stack:v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0",
"method": "stack",
"methodSpecificId": "v0:SZBrgLTLXZL9ZAX8GVNgvZKcU4DJBXkUQr-0",
"parseTree": null,
"parseRuleCount": null
},
"parameters": null,
"parametersMap": {},
"path": "",
"query": null,
"fragment": null,
"parseTree": null,
"parseRuleCount": null
}
},
"methodMetadata": {}
}
Now we know how to recognize and resolve “who’s who” without inherently carrying personally-identifiable information. However, what if we want DID to associate with real-world entities?
Imagine that Alice has a state-issued DID and wants to buy some alcohol without disclosing her real name and precise age.
The answer is to use “verifiable claims” (aka: credentials, attestations).
{
"id": "did:ebfeb1f712ebc6f1c276e12ec21",
"type": ["Identity", "Person"],
"name": "Alice Bobman",
"email": "alice@example.com",
"birthDate": "1985-12-14",
"telephone": "12345678910"
}
{
"id": "http://example.gov/credentials/3732",
"type": ["Credential", "ProofOfAgeCredential"],
"issuer": "https://dmv.example.gov",
"issued": "2010-01-01",
"claim": {
"id": "did:ebfeb1f712ebc6f1c276e12ec21",
"ageOver": 21
}
}
{
"@context": "https://w3id.org/security/v1",
"id": "http://example.gov/credentials/3732",
"type": ["Credential", "ProofOfAgeCredential"],
"issuer": "https://dmv.example.gov",
"issued": "2010-01-01",
"claim": {
"id": "did:ebfeb1f712ebc6f1c276e12ec21",
"ageOver": 21
},
"revocation": {
"id": "http://example.gov/revocations/738",
"type": "SimpleRevocationList2017"
},
"signature": {
"type": "LinkedDataSignature2015",
"created": "2016-06-18T21:19:10Z",
"creator": "https://example.com/jdoe/keys/1",
"domain": "json-ld.org",
"nonce": "598c63d6",
"signatureValue": "BavEll0/I1zpYw8XNi1bgVg/sCneO4Jugez8RwDg/+
MCRVpjOboDoe4SxxKjkCOvKiCHGDvc4krqi6Z1n0UfqzxGfmatCuFibcC1wps
PRdW+gGsutPTLzvueMWmFhwYmfIFpbBu95t501+rSLHIEuujM/+PXr9Cky6Ed
+W3JT24="
}
}
It equals to a JOSE JWT verifiable claim
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2Rtdi
5leGFtcGxlLmdvdiIsImlhdCI6MTI2MjMwNDAwMCwiZXhwIjoxNDgzMjI4ODAwL
CJhdWQiOiJ3d3cuZXhhbXBsZS5jb20iLCJzdWIiOiJkaWQ6ZWJmZWIxZjcxMmVi
YzZmMWMyNzZlMTJlYzIxIiwiZW50aXR5Q3JlZGVudGlhbCI6eyJAY29udGV4dCI
6Imh0dHBzOi8vdzNpZC5vcmcvc2VjdXJpdHkvdjEiLCJpZCI6Imh0dHA6Ly9leG
FtcGxlLmdvdi9jcmVkZW50aWFscy8zNzMyIiwidHlwZSI6WyJDcmVkZW50aWFsI
iwiUHJvb2ZPZkFnZUNyZWRlbnRpYWwiXSwiaXNzdWVyIjoiaHR0cHM6Ly9kbXYu
ZXhhbXBsZS5nb3YiLCJpc3N1ZWQiOiIyMDEwLTAxLTAxIiwiY2xhaW0iOnsiaWQ
iOiJkaWQ6ZWJmZWIxZjcxMmViYzZmMWMyNzZlMTJlYzIxIiwiYWdlT3ZlciI6Mj
F9fX0.LwqH58NasGPeqtTxT632YznKDuxEeC59gMAe9uueb4pX_lDQd2_UyUcc6
NW1E3qxvYlps4hH_YzzTuXB_R1A9UHXq4zyiz2sMtZWyJkUL1FERclT2CypX5e1
fO4zVES_8uaNoinim6VtS76x_2VmOMQ_GcqXG3iaLGVJHCNlCu4
DID
What blockchain developers can do?
Amazon acquiring Whole Foods = Apple’s iPhone beating Palm
Amazon’s Goal
Amazon.com
’s objective is to be the leading online retailer of information-based products and services, with an initial focus on books.Amazon’s Strategy
Tactics: develop grocery services
Book | Grocery |
---|---|
high SKUs = large selection | less SKUs(30k - 50k) |
standardized | vary in quality |
imperishable | perishable |
AmazonFresh’s cost disadvantage
Why can acquiring Whole Food (not doing other things) solve the scale problem?
Primitives model for business with 1) hight fixed costs 2) high returns to scale
AWS Three Layers
Services | Primitives | S3, EC2, RDS, SNS, … |
---|---|---|
Platform | AWS | High Fixed Costs + Returns to Scale |
Infrastructure | Modularized Components | Data center, Servers, Storage, Switches, Bandwidth |
amazon.com
Amazon.com
Three Layers
Services | Packages | FDA, Amazon Pay, … |
---|---|---|
Platform | Fulfillment Centers | High Fixed Costs + Returns to Scale |
Infrastructure | Modularized Suppliers | Manufacturers, 3rd Party Merchants, … |
The insight here is that grocery business has no first-and-best customer.
After fitting in Whole Foods to the big picture, we can see that Amazon is buying more than a retailer - it’s buying a customer.
Amazon.com
Three Layers + Customers
Customers | Whole Foods, Delivery, Restaurants | |
---|---|---|
Services | Groceries | Meat, Fruit, Vegetables, Non-perishables, … |
Platform | Fulfillment Centers | High Fixed Costs + Returns to Scale |
Infrastructure | Modularized Suppliers | Store Brand, Name Brand, Local Suppliers, Regional Suppliers, … |
Now Amazon Grocery Services can serve AmazonFresh and WholeFoods, and then in the future restaurants or whatever can consume it.
]]>There are three reasons-
The kernel of the good strategy
Three examples:
In business, the challenge is usually dealing with change and competition.
In many large organizations, the challenge is often diagnosed as internal.
Amazon’s Flying Wheel
I think the “coherent actions” are very interesting.
A strategy is like designing and throwing a bunch of punching combos: where is the enemy? Where to punch? Where NOT to punch? How to punch? How does your first punch help the second and third ones?
The key idea of this book is the kernel of good strategy. Kernel of strategy = diagnosis + guiding policy + coherent actions
.
A good strategy is surprising but reasonable. For example, when Steve Jobs returned to Apple, he cut the product lines to a few profitable ones. When asked how to compete against the Wintel standard, he just smiled and said, “I am going to wait for the next big thing.”
Another example is Desert Storm. While the media exposure was focusing on the main troops moving slowly in the frontline, another group of military force was moving across the empty desert of southern Iraq as a “left hook.”
Doing everything and thinking everything is important means everything is equally unimportant. Good leaders are supposed to know what to do and what NOT to do.
When we ask what a strategy 101 is, a fundamental answer is SO strategy (Strength + Opportunity).
Shakespeare says in Helmet: there is nothing either good or bad, but thinking makes it so. From a dynamic perspective, strength and weakness are relative. The story of David and Goliath is a good example. People may think that David, the small and the inexperienced, can never fight against Goliath, the giant and the experienced. However, the truth is that David leverages the shepherd’s sling to beat the slow and clumsy Goliath.
Walmart vs. Kmart is another example of finding strength out of where people hardly see. The traditional wisdom of retailing believes that a full-category grocery store has to be placed in an area with the population of at least 100K. However, Walmart can launch its store where there is less population than 100k. How could this happen? It is because that Walmart has a way more efficient supply management system, which makes those stores into an organic network; on the contrary, Kmart does not have close relationships between stores, and they cannot get united, cannot lower inventory costs, and cannot negotiate against suppliers as a whole.
In Cold War, Andrew Marshall designs the strategy to compete against the Soviet Union with comparative advantages - back by the economic and technological advantages of the US, developing things with huge costs but cannot establish actual offensive threats, like increasing the accuracy of missiles or quiet submarines.
Bad strategy is formalism. People mistakenly think the form of the strategy is the most important and cannot embrace the reality. There are four hallmarks to detect bad strategy.
fluff: strategy should not be just a collection of fancy buzzwords.
failure to face the challenge:
mistaking goals for strategy.
bad strategic objectives
What are habits? Do things just like doing no brainers. Businesses cultivating customer habits gain a significant competitive advantage. This book proposes The Hook Model describing how to form a user habit with four steps:
Benefits of Habits
Successful companies build the mind monopoly.
How to identify the product’s habit-forming potential?
What cues people to take action? Triggers.
How to initiate any behavior?
B = MAT (behavior = motivation + ability + trigger)
Back in 2008, companies turning out to have higher total shareholder return(TSR) did these things:
However, slashing costs may hurt the brand and the company moral.
Finally, getting ahead of peers create a huge advantage.
]]>Architecture is the shape of the software system. Thinking it as a big picture of physical buildings.
Together they serve a specific purpose like a hospital is for curing patients and a school is for educating students.
Every software system provides two different values to the stakeholders: behavior and structure. Software developers are responsible for ensuring that both those values remain high.
Software architects are, by virtue of their job description, more focused on the structure of the system than on its features and functions.
Architecture serves the full lifecycle of the software system to make it easy to understand, develop, test, deploy, and operate. The goal is to minimize the human resources costs per business use-case.
The O’Reilly book Software Architecture Patterns by Mark Richards is a simple but effective introduction to these five fundamental architectures.
The layered architecture is the most common in adoption, well-known among developers, and hence the de facto standard for applications. If you do not know what architecture to use, use it.
Examples
Pros and Cons
A state change will emit an event to the system. All the components communicate with each other through events.
A simple project can combine the mediator, event queue, and channel. Then we get a simplified architecture:
Examples
The software’s responsibilities are divided into one “core” and multiple “plugins”. The core contains the bare minimum functionality. Plugins are independent of each other and implement shared interfaces to achieve different goals.
Examples
A massive system is decoupled to multiple micro-services, each of which is a separately deployed unit, and they communicate with each other via RPCs.
Examples
This pattern gets its name from “tuple space”, which means “distributed shared memory". There is no database or synchronous database access, and thus no database bottleneck. All the processing units share the replicated application data in memory. These processing units can be started up and shut down elastically.
Examples: See Wikipedia
operating margin = operating income / net sales
operating income = gross income − (operating expenses + depreciation and amortization)
When it is transiting into a privacy-centric super app, there are three challenges.
Millions of users love the brands and leaders of those unicorns. Those tech stars have everything - except a path to high profits.
In the past 25 years, Three things changed.
Because the unicorns’ markets are contested, margins have not consistently improved, despite fast-rising sales.
The blitzscale philosophy of buying customers at any price is peaking. After the unicorns, a new and more convincing species of startup will have to be engineered.
Alas! Andrew Grove says - Success breeds complacency. Complacency breeds failure. Only the paranoid (who embrace change) survive.
]]>content analysis = derive intermediate data from raw articles and user behaviors.
Take articles for example. To model user interests, we need to tag contents and articles. To associate a user with the interests of the “Internet” tag, we need to know whether a user reads an article with the “Internet” tag.
We do it for the reason of …
Here is an example of “article features” page. There are article features like categorizations, keywords, topics, entities.
What are the article features?
Semantic Tags: Human predefine those tags with explicit meanings.
Implicit Semantics, including topics and keywords. Topic features are describing the statistics of words. Certain rules generate keywords.
Similarity. Duplicate recommendation once to be the most severe feedbacks we get from our customers.
Time and location.
Quality. Abusing, porn, ads, or “chicken soup for the soul”?
We divide features of semantic tags into three levels:
Why dividing into different levels? We do this so that they can capture articles in different granularities.
Categorizations and concepts are sharing the same technical infrastructure.
Why do we need semantic tags?
Classification hierarchy
Classifiers:
A blockchain is an incorruptible distributed ledger that is…
Hardware: computer resources = computing + networking + storage
Basic Utils: P2P network + crypto + data storage w/ db or filesystem
Ledger: chain of data blocks + domain-specific data models
Consensus: write first consensus later (PoW/PoS/DPoS) / consensus first write later (PBFT)
Smart Contract: limited program running on the blockchain
API: RPC + SDK
dApps: 1) transfer of values 2) data certification 3) data access control
DevOps: deployment, operations, metrics, logs
Financial Services
Health care
Public sector
Energy and resources
Technology, media, and telecom
Consumer and industrial products
We are finding the best function
below to maximize user satisfaction
.
user satisfaction = function(content, user profile, context)
Measurable Goals, e.g.
Hard-to-measurable Goals:
It is a typical supervised machine learning problem to find the best function
above. To implement the system, we have these algorithms:
A world-class recommendation system is supposed to have the flexibility to A/B-test and combine multiple algorithms above. It is now popular to combine LR and DNN. Facebook used both LR and GBDT years ago.
Correlation, between content’s characteristic and user’s interest. Explicit correlations include keywords, categories, sources, genres. Implicit correlations can be extract from user’s vector or item’s vector from models like FM.
Environmental features such as geo location, time. It’s can be used as bias or building correlation on top of it.
Hot trend. There are global hot trend, categorical hot trend, topic hot trend and keyword hot trend. Hot trend is very useful to solve cold-start issue when we have little information about user.
Collaborative features, which helps avoid situation where recommended content get more and more concentrated. Collaborative filtering is not analysing each user’s history separately, but finding users’ similarity based on their behaviour by clicks, interests, topics, keywords or event implicit vectors. By finding similar users, it can expand the diversity of recommended content.
They are implemented in the following steps:
It is impossible to predict all the things with the model, considering the super-large scale of all the contents. Therefore, we need recall strategies to focus on a representative subset of the data. Performance is critical here and timeout is 50ms.
Among all the recall strategies, we take the InvertedIndex<Key, List<Article>>
.
The Key
can be topic, entity, source, etc.
Tags of Interests | Relevance | List of Documents |
---|---|---|
E-commerce | 0.3 | … |
Fun | 0.2 | … |
History | 0.2 | … |
Military | 0.1 | … |
We are finding the best function
below to maximize user satisfaction
.
user satisfaction = function(content, user profile, context)
Measurable Goals, e.g.
Hard-to-measurable Goals:
It is a typical supervised machine learning problem to find the best function
above. To implement the system, we have these algorithms:
A world-class recommendation system is supposed to have the flexibility to A/B-test and combine multiple algorithms above. It is now popular to combine LR and DNN. Facebook used both LR and GBDT years ago.
Correlation, between content’s characteristic and user’s interest. Explicit correlations include keywords, categories, sources, genres. Implicit correlations can be extract from user’s vector or item’s vector from models like FM.
Environmental features such as geo location, time. It’s can be used as bias or building correlation on top of it.
Hot trend. There are global hot trend, categorical hot trend, topic hot trend and keyword hot trend. Hot trend is very useful to solve cold-start issue when we have little information about user.
Collaborative features, which helps avoid situation where recommended content get more and more concentrated. Collaborative filtering is not analysing each user’s history separately, but finding users’ similarity based on their behaviour by clicks, interests, topics, keywords or event implicit vectors. By finding similar users, it can expand the diversity of recommended content.
They are implemented in the following steps:
It is impossible to predict all the things with the model, considering the super-large scale of all the contents. Therefore, we need recall strategies to focus on a representative subset of the data. Performance is critical here and timeout is 50ms.
Among all the recall strategies, we take the InvertedIndex<Key, List<Article>>
.
The Key
can be topic, entity, source, etc.
Tags of Interests | Relevance | List of Documents |
---|---|---|
E-commerce | 0.3 | … |
Fun | 0.2 | … |
History | 0.2 | … |
Military | 0.1 | … |
content analysis = derive intermediate data from raw articles and user behaviors.
Take articles for example. To model user interests, we need to tag contents and articles. To associate a user with the interests of the “Internet” tag, we need to know whether a user reads an article with the “Internet” tag.
We do it for the reason of …
Here is an example of “article features” page. There are article features like categorizations, keywords, topics, entities.
What are the article features?
Semantic Tags: Human predefine those tags with explicit meanings.
Implicit Semantics, including topics and keywords. Topic features are describing the statistics of words. Certain rules generate keywords.
Similarity. Duplicate recommendation once to be the most severe feedbacks we get from our customers.
Time and location.
Quality. Abusing, porn, ads, or “chicken soup for the soul”?
We divide features of semantic tags into three levels:
Why dividing into different levels? We do this so that they can capture articles in different granularities.
Categorizations and concepts are sharing the same technical infrastructure.
Why do we need semantic tags?
Classification hierarchy
Classifiers:
result = pairs.map((pair) => (morePairs)).reduce(somePairs => lessPairs)
in a distributed system.To serve requirements above with commodity machines, the steaming framework use distributed systems in these architectures…
Framework | Storm | Storm-trident | Spark | Flink |
---|---|---|---|---|
Model | native | micro-batch | micro-batch | native |
Guarentees | at-least-once | exactly-once | exactly-once | exactly-once |
Fault-tolerance | Record-Ack | record-ack | checkpoint | checkpoint |
Overhead of fault-tolerance | high | medium | medium | low |
latency | very-low | high | high | low |
throughput | low | medium | high | high |
Leveraging user and device data during user login to fight against
ATOs ranking from easy to hard to detect
Semi-supervised learning = unlabeled data + small amount of labeled data
Why? better learning accuracy than unsupervised learning + less time and costs than supervised learning
Challenges
Architecture
Brand / Advertiser: individuals or organizations who want to publish advertising messages to the customers.
Agency: they help the brand to interact with the rest of the ecosystem and manage the whole lifecycle of the advertising messages, including planning, creating, and distributing ad campaigns.
Trading Desk: It streamlines the media buying process.
Demand-side Platform (DSP): it automates online ad inventory and buying, helping agencies to manage accounts across different accounts and campaigns through one platform.
Data-management Platform (DMP)
Ad Exchange / Real-time Bidding (RTB): It matches ads suppliers with buyers.
Ad Network: It aggregates publisher inventory and sells it to advertisers.
Supply Side Platform (SSP): It monitors the entire ads inventory and suggest prices for ad space.
Publisher: Ad-space owners like website operators.
Unfortunately, one management style does not fit all the people in all the scenarios. A fundamental variable to find the best management style is task-relevant maturity (TRM) of the subordinates.
TRM | Effective Management Style |
---|---|
low | structured; task-oriented; detailed-oriented; instruct exactly “what/when/how mode” |
medium | Individual-oriented; support, “mutual-reasoning mode” |
high | goal-oriented; monitoring mode |
A person’s TRM depends on the specific work items. It takes time to improve. When TRM reaches the highest level, the person’s both knowledge-level and motivation are ready for her manager to delegate work.
The key here is to regard any management mode not as either good or bad but rather as effective or not effective.
]]>By and large, anytime you change what people regularly use in a product, they will always throw an uproar. This happens to almost every release of products like Gmail, YouTube, iPhone, etc.
The product changes may turn out to be good or bad ones.
]]>Answer: for reducing human resources costs per feature.
Mobile developers evaluate the architecture in three dimensions.
Distribution of Responsibility | Testability | Ease of Use | |
---|---|---|---|
Tight-coupling MVC | ❌ | ❌ | ✅ |
Cocoa MVC | ❌ VC are coupled | ❌ | ✅⭐ |
MVP | ✅ Separated View Lifecycle | ✅ | Fair: more code |
MVVM | ✅ | Fair: because of View’s UIKit dependant | Fair |
VIPER | ✅⭐️ | ✅⭐️ | ❌ |
For example, in a multi-page web application, page completely reloaded once you press on the link to navigate somewhere else. The problem is that the View is tightly coupled with both Controller and Model.
Apple’s MVC, in theory, decouples View from Model via Controller.
Apple’s MVC in reality encourages massive view controllers. And the view controller ends up doing everything.
It is hard to test coupled massive view controllers. However, Cocoa MVC is the best architectural pattern regarding the speed of the development.
In an MVP, Presenter has nothing to do with the life cycle of the view controller, and the View can be mocked easily. We can say the UIViewController is actually the View.
There is another kind of MVP: the one with data bindings. And as you can see, there is tight coupling between View and the other two.
It is similar to MVP but binding is between View and View Model.
There are five layers (VIPER View, Interactor, Presenter, Entity, and Routing) instead of three when compared to MV(X). This distributes responsibilities well but the maintainability is bad.
When compared to MV(X), VIPER
O(log n)
to hash-based ones of O(1)
to read and writeThere are various cache policies like read-through/write-through(or write-back), and cache-aside. By and large, Internet services have a read to write ratio of 100:1 to 1000:1, so we usually optimize for read.
In distributed systems, we choose those policies according to the business requirements and contexts, under the guidance of CAP theorem.
When a cache does not support native read-through and write-through operations, and the resource demand is unpredictable, we use this cache-aside pattern.
There are still chances for dirty cache in this pattern. It happens when these two cases are met in a racing condition:
For the Photos application most of this metadata, such as permissions, is unused and thereby wastes storage capacity. Yet the more significant cost is that the file’s metadata must be read from disk into memory in order to find the file itself. While insignificant on a small scale, multiplied over billions of photos and petabytes of data, accessing metadata is the throughput bottleneck.
Eliminates the metadata overhead by aggregating hundreds of thousands of images in a single haystack store file.
index file (for quick memory load) + haystack store file containing needles.
index file layout
haystack store file
Upload
Download
]]>Conway’s law says structures of software systems are copies of the organization structures.
Monolithic Service | Micro Services | |
---|---|---|
Productivity, when teams and codebases are small | ✅ High | ❌ Low |
Productivity, when teams and codebases are large | ❌ Low | ✅ High (Conway’s law) |
Requirements on Engineering Quality | ❌ High (under-qualified devs break down the system easily) | ✅ Low (runtimes are segregated) |
Dependency Bump | ✅ Fast (centrally managed) | ❌ Slow |
Multi-tenancy support / Production-staging Segregation | ✅ Easy | ❌ Hard (each individual service has to either 1) build staging env connected to others in staging 2) Multi-tenancy support across the request contexts and data storage) |
Debuggability, assuming same modules, metrics, logs | ❌ Low | ✅ High (w/ distributed tracing) |
Latency | ✅ Low (local) | ❌ High (remote) |
DevOps Costs | ✅ Low (High on building tools) | ❌ High (capacity planning is hard) |
Combining monolithic codebase and micro services can bring benefits from both sides.
The key is to have an async design, because payment systems usually have a very long latency for ACID transactions across multiple systems.
Today, let us focus on how to operate the cycle system.
Everyday, repeat the following process.
Start a new day with planning.
Create a schedule. Fill in with meetings and appointments. So we can estimate the time left for working on tasks.
Create to-do list. Remember to count the hours in those tasks because this helps you estimate the workload.
Prioritize and Reschedule.
Work the plan
Finish the day with a review and reschedule
How to deal with new tasks during the day?
Never miss a meeting and show up on time. It demonstrates your responsibility reliability. And the secret is to always use your calendar and don’t trust your brain to agree an appointment without checking your calendar. If you are going to be late or miss an appointment, always call.
The calendar is the place to put events fixed to happen in a certain time range. They can be:
Adjust yourself according to your personal rhythms and your company’s rhythms. Most important (high-impact) work should be scheduled to peak hours you can focus with the highest energy level.
Balance is important. Work, family life, social life, volunteer work, personal projects, sleep are all important and can be scheduled well with a centralized calendar.
You will achieve more if you set goals. And setting a goal without working toward it is better than having no goals at all. If you haven’t determined what your goals are, you cannot spot the few opportunities that do cross your path by chance.
The secret lies in writing down your goals. Apply the SMART principles during the goal-setting. (Specific, Measurable, Assignable, Relevant, Time-based). However, stretching the goals and challenging yourself is very important to boost personal growth. Keep those goals in two dimensions:
Then develop strategies for those goals -
When working on those goals over time, be sure to keep up with the system. Periodically review them per month, quarter, and year, which includes:
Finally, use your calendar for this repeated “Goal & Next Step Review.”
]]>Auth0 | Okta | Amazon Cognito | onelogin | Firebase Authentication | |
---|---|---|---|---|---|
Send Welcome Email after Signup | Template provided | ❌ | ❌ | ❌ | event handler provide but need email vendor integration |
Customer Type | B2C, B2B, B2E | B2C, B2B, B2E | ? | B2E | ❌ |
SSO | ✅ | ✅ | ✅ | ✅ | ? |
MFA | Push Notification, SMS | Authenticator, SMS, Voice Call, Security Question | SMS, Authenticator | Push, SMS, Authenticator | SMS |
Social Login / Public Identity Providers | ✅ | ✅ | ✅ | ✅ | ✅ |
Login Rules Engine / Policy | ✅ | ✅ | ❌ | ✅ | ❌ |
RBAC / Group-based | ✅ | ✅ | ✅ | ✅ | ❌ |
Cross-platform SDK | Web, Mobile, Native | Web(Angular, Node.js, React, PHP, Java, .NET), Mobile(iOS, Android), Native (Java, .NET), Machine-to-machine | ✅ | raw examples | iOS, Android, Web, C++, Unity, |
Industry Standards | SAML, OpenID Connect, JWT, OAuth2.0, OAuth1.0a, WS-Federation, OpenID | SAML identity provider | OAuth2.0, SAML2.0, OpenID Connect | SAML 1.1 and 2.0 WS-Federation 2005 SCIM 1.1 and 2.0 OAuth 1.0 and 2.0 OpenID Connect 1.0 JSON Web Token (JWT) Integrated Windows Authentication (IWA) | ❌ |
Analytics | ✅ | ✅ | aws Pinpoint | ❌ | ❌ |
General SLA | 99.95% | 99.97% | ❌ | 99.98% | ❌ |
Passwordless | Touch ID, Email Magic Link, SMS | ❌ | ❌ | ❌ | ❌ |
Anomaly Detection | 1. Brute-force ProtectionLimit the amount of signups and failed logins from a suspicious IP address.2. Breached-password Detection. Detects login attempts with credentials that have been known to be breached. | Risk-based authentication | ❌ | Risk-based authentication | ❌ |
Anomaly Detection Reactions | Email notificationBlock IP | ❌ | ❌ | ❌ | ❌ |
Providing User Profile / Directory Store | ✅very extensive | ✅W/ Management Metrics: Total Users, Authentications, Failed Logins, System Log | ✅access configured by Apps | ✅ | ✅very limited fields |
Workflows - Email address verification | ❌ | ❌ | ❌ | ❌ | ✅ |
Workflows - Email address change | ❌ | ❌ | ❌ | ❌ | ✅ |
Workflows - Forgot password | ✅limited page customization | ✅email templates | ✅email/SMS template | ✅no UI customization | ✅ |
Workflows - Lockout Self-Service | ❌ | ✅ | ❌ | ❌ | ❌ |
AD/LDAP integration | ✅ | ✅ | AD | AD | ❌ |
Compliance | SOC 2 Type II, EU-US Privacy Shield Framework, HIPAA, OpenID Connect | HIPAA, EU, and FED compliance | PCI DSS Compliance and is HIPAA Eligible | ASSURANCE PROGRAMS: SOC 2 Type 2 SOC 1 Type 2 ISO 27017:2015 ISO 27018:2014 ISO 27001:2013 SECURITY PROGRAMS: Skyhigh Enterprise-Ready CSA STAR PRIVACY PROGRAMS: TRUSTe Certified Privacy U.S. Privacy Shield GDPR EU Model Contract Clauses VULNERABILITY MANAGEMENT: Penetration Tests Network Scans Bug Bounty Program OTHER INITIATIVES: HIPAA FFIEC / GLBA NIST Cybersecurity Framework G-Cloud FERPA | ❌ |
User Devices Management | ✅ | ✅ | ✅ | ✅ | ❌ |
You are already familiar with many of the tricks:
Figures of speech - Making words presented differently by repetition, substitution, sound, and wordplay. Making words sound differently by skipping, swapping, etc.
Repeated first word: use a lot of “and” to start the sentence while thinking what to say.
Multiple yoking
Idiom
Self-answering question
Tropes: swapping
Win the intelligent audience by twisting the expression. For example, adding a surprising end. e.g.
The mighty ABBA sentences (chiasmus), e.g.
Or even more, inserting a pun into a chiasmus. e.g.
Dialysis: Either… or… e.g., George W Bush: you’re either with us, or you’re with the terrorists.
Antithesis: Not… but… e.g., The success of our economy has always depended not just on the size of our gross domestic product, but on the reach of our prosperity.
Inventing new words is dangerous in high school or a government agency. However, it is impactful so we would better use it wisely.
examples of inventing
Style | Target Characteristics | Pursuader’s Strategy |
---|---|---|
Charismatic | easily enthralled but make decisions based on balanced info Emphasize bottom-line results |
Focus on results be straight-forward benefits w/ visual aids keyword: proven, actions, easy, clear |
Thinker (Xing Wang) | toughest to persuade needs extensive detail |
presents market research, surveys, cost/benefit analysis. keyword: quality, numbers, expert, proof |
Skeptic | challenge everything and make decisions based on gut feelings | establish credibility with endorsements from someone they trust. keyword: grasp, power, suspect, trust. |
Follower | rely on past decisions late adopter |
Use testimonials to prove low risks. present innovative but proven solutions. keyword: expertise, similar to, innovate, previous. |
Controller | unemotional, analytical only implements own ideas |
present highly structured arguments make listener own the idea. avoid aggressive advocacy. keyword: facts, reason, power, just do it. |
By building enduring engagement, in three levels:
Growing engaged users: focus on growing users completing the core action.
Retaining users: product should get better the more it’s used. Users have more to lose by leaving the product.
Self-perpetuating: As users engage, they create virtuous loops in the product.
Why should we learn time management from a system administrator? Thomas Limoncelli says “I’m a system administrator! I manage chaos for a living!”
Your customers value your ability to follow through more than they appreciate any other skill you have. Nothing ruins your reputation like agreeing to do something and forgetting to do it.
The Cycle system. It is called cycle because it repeats every day and the output of one day is the input to the next. It uses these three tools -
Keep them in one single place and sync across all devices. And don’t trust your brain for remembering and prioritizing and scheduling tasks.
Routines, for example, can be
Mantras, for example, can be
How to develop routines? Try to find …
Repeated events that aren’t scheduled.
Maintenance tasks.
Relationships and career networking. Relationships require maintenance and are also similar to gardening (they grow if you work diligently, starve if they are ignored, and die if they get too much attention). There are four types of people to maintain relationships
When procrastinating takes longer than action.
Things you forget often.
Inconsequential or low-priority tasks that can be skipped occasionally but shouldn’t be.
Developing new skills.
Keeping up-to-date by reading.
We all know that the South Pole is located at the south-most of the earth, namely the 90°S in latitude. A typical exploratory plan was that expeditions started from 82°S, went to the South Pole, and then came back alive.
Two teams were competing against each other for the first place - the Amundsen team with 5 members and the Scott team with 17 members. Which one do you guess will win this competition? Of course, more people do not necessarily mean more chances of success.
They set off almost at the same time. As the same with all the competitions ever happening in the world, it is quite intensive. When there are great opportunities, there are no reasons that only you can see it; unquestionably, there are a lot of people can see it. Both teams prepared well around October 1911 at the periphery of the Antarctic Circle, and they were racing and rushing for the last distance.
The result was like this – the Amundsen team planted the Norwegian flag at the South Pole first in the following two months, namely December 15th, 1911. However, the Scott team was late for more than one month, though they started almost at the same time and had more team members… What does this mean?
It means the difference between success and failure. Amundsen team was remembered as the first one reaching the South Pole in human history, and the winner takes all the honors. Unfortunately, the Scott team suffered the same challenges but was just late. No one would remember the second place, but we all remember the first one.
This story was not as simple as the above. There were even more - you should not just go to the South Pole; you should come back alive as well. The Amundsen team went there first and came back to the base smoothly.
On the other hand, the Scott team was late and failed to win the glory. Even worse, because of being late, the weather became awful during the way back. People left behind in increasing numbers. Finally, none of them survived. This team, these 17 people, failed to achieve the victory and perished as a whole. It is the difference between death and life.
Today, we can say that they are risking more than us the entrepreneurs. The bet was bigger and thriller than we could imagine. Why was this difference more than just between success and failure, but actually between death and life? Researching the causing facts gives us the insights.
First, exploring the South Pole is not just about people; it is also about the supplies. Researchers analyzed afterward and found the vast difference in the preparation. The Amundsen team prepared three tons of supplies though they had fewer team members. The Scott team made only one ton of supplies, though they had more team members.
Is one ton of supplies adequate? If you make no mistakes, completely no mistakes, then that is just enough. It is horrible that things are perfect in theory and you plan with a tight schedule of resources. People come across unexpected scenarios all the time in reality. People get lost in the wilderness all the time when exploring. People make inevitable mistakes all the time under stress. The fact is that a plan without any slack leads to grave danger.
On the contrary, the Amundsen team did a great job on this. They had only 5 people but prepared three tons of supplies. The surplus in resources made them more fault-tolerant and well-prepared for the unexpected challenges.
It is a considerable difference whether the resource is abundant enough and whether the team leaves room for making mistakes.
In fact, both teams were competing in the same environment, but they delivered two fundamentally different results, which is well worth reaching.
In one word, the success of the Amundsen team is due to making progress 30 kilometers per day no matter what the weather is. In extreme environments, you do the best. More importantly, you do the best in a sustainable way.
Unfortunately, the Scott team was less-disciplined according to their logs. They could advance 40 to 60 kilometers in one day if the weather were pleasant. However, when the weather was terrible, they were bad-tempered, they cursed the bad luck, and they stayed in the tent for the entire day.
In retrospective, this might be the most significant difference. The difference is that no matter how bad the weather is, keep moving 30 kilometers a day and then you can reach the South Pole and then come back alive.
Why am I telling this story? It precisely resembles the intense competition today we are facing in the Internet era. People may say it is the winter of the market and things are getting worse. It is the same with the awful weather 100 years ago in the South Pole. What we can do to survive is like the Amundsen team - making plans with slackness to prepare for the unexpected; leaving room for making mistakes; and most importantly, making progress 30 kilometers per day, and no matter how bad the weather is.
]]>I may say that this is the greatest factor—the way in which the expedition is equipped—the way in which every difficulty is foreseen, and precautions taken for meeting or avoiding it. Victory awaits him who has everything in order—luck, people call it. Defeat is certain for him who has neglected to take the necessary precautions in time; this is called bad luck.
- from The South Pole, by Roald Amundsen
The shift from a culture of humility to the culture of “big me” encourages people to nurture Adam I.
In the times of humility, elder George Bush, he resisted speaking about himself and crossed out the word “I” instinctively in the speech text. And the speechwriter would beg him: You’re running for president. You’ve got to talk about yourself. Finally, they did cow him into doing so. However, the next day he’d get a call from his mother. “George, you’re talking about yourself again.”
This shift may be caused by the increasingly tremendous benefits brought by fame. (Thanks for the mass media and then the Internet.) Since the culture prefers people who self-promote, people become more narcissistic than ever.
Pride is the desire to see yourself as superior to everybody else. Humility is the self-confrontation of weakness. Thus it is painful and takes efforts to build the character with humility. You are not alone. Kant says we are all made from crooked timber.
How to improve my humility? Reflect on errors by asking questions and develop strategies to act differently next time. Did I make mistakes today? Am I putting my loves in disorder? Am I not fully present for people who are asking my advice or revealing some vulnerability? Am I more interested in making a good impression than in listening to other people in depth?
The author David Brooks believes - only Adam II can experience deep satisfaction in life. I think that regarding SWOT analysis, in addition to strength-opportunity strategy, this book advises another direction for actions: weaknesses-threats strategy.
]]>To work with Context Provider,
To work with Deliberative,
To work with Commander,
To work with Communicator,
To work with Competitor,
To work with Connector,
To work with Activator
To work with Adaptor
To work with Analyzer
To work with Arranger
Total time used = time delayed + time wasted
Time wasted = resuming time for context switch + recovering from mistakes
What is the best friend of productivity?
Focus.
How to focus?
An entry-level work we can do is to de-clutter our brain by recording unrelated ideas at somewhere outside of the brain. Use our task tracker extensively.
Be aware of our stress and sleep level. If we are tired or under much pressure, sleep well and multitask less.
Creating an un-distracting environment.
Deal with interruptions effectively.
Having difficulty falling asleep?
Keep a pad of paper and a pen next to your bed. When something is keeping you awake, write it down and try falling asleep again. Worry keeps us awake because we’re trying to remember to do something about what’s worrying us. Anger keeps us awake because we’re trying to remember to stay angry!
Staying focus and not being rude
And SAs have to deal with those issues even more often, as Thomas Limoncelli says:
Management judges an SA by whether projects get done. Customers, however, judge you by whether you are available to them.
Here are SAs’ principles of time management.
Interruption is the archenemy of productivity.
unify all time management information into one place.
conserve the brain power for things important
don’t make yourself think to manage time; instead, develop routines, habits, and mantras.
keep focus during the project time. It takes discipline though.
manage your social life with same tools.
In-memory HashMap<Key, <FildId, ValueOffset, ValueSize, Timestamp>>
Data file layout
|crc|timestamp|key_size|value_size|key|value|
...
Delete: get the location by the in-memory hashmap, if it exists, then go to the location on the disk to set the value to a magic number.
Get: get the location by the in-memory hashmap, and then go to the location on the disk for the value.
Put: append to the active data file and update the in-memory hash map.
Periodical compaction strategies
Copy latest entries: In-memory hashmap is always up-to-date. Stop and copy into new files. Time complexity is O(n) n is the number of valid entries.
Scan and move: foreach entry, if it is up-to-date, move to the tail of the validated section. Time complexity is O(n) n is the number of all the entries.
Following up questions
I’m not sure it’s for you, but …
Open-minded
What do you know about
How would you feel if?
Just imagine
When would be a good time?
I’m guessing you haven’t got around to
Simple Swaps
As I see it, you have three options … Of those three options, what’s going to be easier for you?
There are two types of people, …
I bet you’re a bit like me
If … then …
Don’t worry
Most people
The Good News
What happens next
What makes you say that
before you make your mind up
Target a few hundred or a thousand key people, not millions.
Do not target all people - target the right people
e.g., Uber offered free rides for Austin’s SXSW conference for several years, which attracts thousands of tech-obsessed, high-income young adults.
Hacks
Stunts
Focusing on new user sign-ups (acquisition) instead of awareness.
Growth Techniques = marketing + engineering
Product/market fit is the degree to which a product satisfies a strong market demand.
Start with MVP and evolve with feedbacks
Use data and information to back PMF.
Understand the needs of the customers as early as possible
Develop answers with the Socrates method
Hot Standby: Keep two active systems undertaking the same role. Data is mirrored in near real time, and both systems will have identical data.
Warm Standby: Keep two active systems but the secondary one does not take traffic unless the failure occurs.
Checkpointing (or like Redis snapshot): Use write-ahead log (WAL) to record requests before processing. Standby node recovers from the log during the failover.
Active-active (or all active): Keep two active systems behind a load balancer. Both of them take in parallel. Data replication is bi-directional.
]]>Assumptions: I generally don’t include these assumptions in the initial problem presentation. Good candidates will ask about scale when coming up with a design.
1B new URLs per day, 100B entries in total the shorter, the better show statics (real-time and daily/monthly/yearly)
http://blog.codinghorror.com/url-shortening-hashes-in-practice/
Choice 1. md5(128 bit, 16 hex numbers, collision, birthday paradox, 2^(n/2) = 2^64) truncate? (64bit, 8 hex number, collision 2^32), Base64.
Choice 2. Distributed Seq Id Generator. (Base62: a~z, A~Z, 0~9, 62 chars, 62^7), sharding: each node maintains a section of ids.
MySQL(10k qps, slow, no relation), KV (100k qps, Redis, Memcached)
A great candidate will ask about the lifespan of the aliases and design a system that purges aliases past their expiration.
Q: How will shortened URLs be generated?
Q: How to store the mappings?
Q: How to implement the redirect servers?
Q: How are click stats stored?
Q: How will the aggregation tier be partitioned?
Q: How to prevent visiting restricted sites?
To solve three problems introduced by big data
e.g. problems with scaling a pageview service in a traditional way
The key point is that X-axis dimension alone of the AKF scale cube is not good enough. We should introduce Y-axis / functional decomposition as well. Lambda architecture tells us how to do it for a data system.
If we define a data system as
Query = function(all data)
Then a lambda architecture is
batch view = function(all data at the batching job's execution time)
realtime view = function(realtime view, new data)
query = function(batch view. realtime view)
Lambda architecture = CQRS (batch layer + serving layer) + speed layer
]]>To reduce redundancy and improve consistency, people follow 3NF when designing database schemas:
What if we want to eliminate single point of failure? What if the dataset is too large for one single machine to hold? For MySQL, the answer is to use a DB proxy to distribute data, either by clustering or by sharding
Clustering is a decentralized solution. Everything is automatic. Data is distributed, moved, rebalanced automatically. Nodes gossip with each other, (though it may cause group isolation).
Sharding is a centralized solution. If we get rid of properties of clustering that we don’t like, sharding is what we get. Data is distributed manually and does not move. Nodes are not aware of each other.
]]>To optimize the read performance, denormalization is introduced by adding redundant data or by grouping data. These four categories of NoSQL are here to help.
The abstraction of a KV store is a giant hashtable/hashmap/dictionary.
The main reason we want to use a key-value cache is to reduce latency for accessing active data. Achieve an O(1) read/write performance on a fast and expensive media (like memory or SSD), instead of a traditional O(logn) read/write on a slow and cheap media (typically hard drive).
There are three major factors to consider when we design the cache.
Out-of-box choices: Redis/Memcache? Redis supports data persistence while Memcache does not. Riak, Berkeley DB, HamsterDB, Amazon Dynamo, Project Voldemort, etc.
The abstraction of a document store is like a KV store, but documents, like XML, JSON, BSON, and so on, are stored in the value part of the pair.
The main reason we want to use a document store is for flexibility and performance. Flexibility is achieved by the schemaless document, and performance is improved by breaking 3NF. Startup’s business requirements are changing from time to time. Flexible schema empowers them to move fast.
Out-of-box choices: MongoDB, CouchDB, Terrastore, OrientDB, RavenDB, etc.
The abstraction of a column-oriented store is like a giant nested map: ColumnFamily<RowKey, Columns<Name, Value, Timestamp>>.
The main reason we want to use a column-oriented store is that it is distributed, highly-available, and optimized for write.
Out-of-box choices: Cassandra, HBase, Hypertable, Amazon SimpleDB, etc.
As the name indicates, this database’s abstraction is a graph. It allows us to store entities and the relationships between them.
If we use a relational database to store the graph, adding/removing relationships may involve schema changes and data movement, which is not the case when using a graph database. On the other hand, when we create tables in a relational database for the graph, we model based on the traversal we want; if the traversal changes, the data will have to change.
Out-of-box choices: Neo4J, Infinitegraph, OrientDB, FlockDB, etc.
]]>How often do you have an opportunity to listen to your customers describe their problems? The answer is probably NEVER if you are an employee positioned in a non-marketing department of a large company.
If you do have the chance, two things lie at the core of the buyer persona concept,
KYC (knowing your customer) is not easy. e.g. iPhone 3G was not selling well in Japan in 2008. Japanese customers were accustomed to using phones to shoot videos / pay with debit card chips / train pass chips.
A generic buyer profile cannot make marketer understand exactly what determine’s the buyer’s buying decision. Marketers are just guessing based on demographics (age, income, marital status, education) or psychographics (personality, values, lifestyles, opinions).
The buyer profile can still give some obvious answers though. e.g. reaching CFO via an email campaign is so difficult. Emphasizing the spaciousness of the car’s cargo for a large dog is not useful for a busy woman that only raises goldfishes.
Rather than guessing, the most effective way to build buyer personas is to interview buyers who have previously weighed their options, considered or rejected solutions and made a decision similar to the one you want to influence.
Buyer persona = buyer profile (who will buy) + buyer insights (when/how/why to buy)
]]>To improve communication quality by valuing everyone’s needs. Judgments and violence are tragic expressions of unmet needs.
e.g. Are you feeling … because you need …?
LTV:CAC Ratio helps you determine how much you should be spending to acquire a customer, so that you can achieve sustainable growth.
1:1
= lose money the more you sell3:1
or better = good.5:1
or higher = under-investing in marketingUsecases
False positive matches are possible, but false negatives are not – in other words, a query returns either “possibly in set” or “definitely not in set”. Elements can be added to the set, but not removed (though this can be addressed with a “counting” bloom filter); the more elements that are added to the set, the larger the probability of false positives.
Usecases
JSON RPC | GraphQL | REST | gRPC | |
---|---|---|---|---|
Usecases | Etherum | Github V2, Airbnb, Facebook BFF / API Gateway | Swagger | High performance, Google, internal endpoints |
Single Endpoint | ✅ | ✅ | ❌ | ✅ |
Type System | ✅ as weak as JSON No uint64 |
✅ No uint64 |
✅ w/ Swagger No uint64 |
✅ has uint64 |
Tailored Results | ❌ | ✅ | ❌ | ❌ |
Batch nested queries | ❌ | ✅ | ❌ | ❌ |
Versioning | ❌ | Schema Extension | Yes, w/ v1/v2 route s | Field Numbers in protobuf |
Error Handling | Structured | Structured | HTTP Status Code | Structured |
Cross-platform | ✅ | ✅ | ✅ | ✅ |
Playground UI | ❌ | GraphQL Bin | Swagger | ❌ |
Performance tracing | ? | Apollo plugin | ? | ? |
caching | No or HTTP cache control | Apollo plugin | HTTP cache control | Native support not yet. but still yes w/ HTTP cache control |
Problem | Lack of community support and toolchainBarrister IDL | 42.51 kb client-side bundle size | Unstructured with multiple endpoints. awful portability. | Grpc-web dev in progress140kb JS bundle. Compatibility issues: not all places support HTTP2 and grpc dependencies |
Those who are NOT sources of an economic moat
Those who are sources of an economic moat
Jerry Neumann: A taxonomy of moats.
]]>An interview is a process for workers to find future co-workers, during which they are finding signals to answer the following three key questions:
None of the critical questions above can be answered without good communication. Your job will be taken away by people who can talk better than you.
leaders = visionaries who inspire self-sacrifice
. A leader does not exist without the ability to persuade.Its abstraction is a queue and it features
It can be applied to
Kafka is using zero copy in which that CPU does not perform the task of copying data from one memory area to another.
Without zero copy:
With zero copy:
Looking from outside, producers write to brokers, and consumers read from brokers.
Data is stored in topics and split into partitions which are replicated.
How to serialize data? Avro
What is its network protocol? TCP
What is a partition’s storage layout? O(1) disk read
In-sync replica (ISR) protocol. It tolerates (numReplicas - 1) dead brokers. Every partition has one leader and one or more followers.
Total Replicas = ISRs + out-of-sync replicas
Jun Rao says it is CA, because “Our goal was to support replication in a Kafka cluster within a single datacenter, where network partitioning is rare, so our design focuses on maintaining highly available and strongly consistent replicas.”
However, it actually depends on the configuration.
Out of the box with default config (min.insync.replicas=1, default.replication.factor=1) you are getting AP system (at-most-once).
If you want to achieve CP, you may set min.insync.replicas=2 and topic replication factor of 3 - then producing a message with acks=all will guarantee CP setup (at-least-once), but (as expected) will block in cases when not enough replicas (<2) are available for particular topic/partition pair.
In the battle ground, if you don’t sleep, you’ll burn out pretty quickly. You’ll make bad decisions. You’ll let people down and become a liability.
Principle: Relax the whole body one part by another and don’t think.
Before TAO, use cache-aside pattern
Social graph data is stored in MySQL and cached in Memcached
3 problems:
To solve those problems, we have 3 goals:
Efficiency at scale and reduce read latency
Write timeliness
Read availability
Read failover
]]>The answer is Anna Karenina principle. Tolstoy opens Anna Karenina by observing: “All happy families are alike; each unhappy family is unhappy in its own way.” Business is the opposite. All happy companies are different: each one earns a monopoly by solving a unique problem. All failed companies are the same: they failed to escape competition.
If a startup does not innovate but copy a product or service from the market leader, and the startup is targeting the same market, then people will not buy it because people are probably customers of the market leader already. Why do people buy the same thing for twice if their needs are fulfilled already?
]]>Point 4 is the insight here - referencing each other is key to the marketing success. If two people buy the same product for the same reason but have no way they could reference each other, they are not part of the same market. They are in different market segments.
]]>How to keep users’ viewing data in scale (billions of events per day)?
Here, viewing data means…
The viewing service has two tiers:
stateful tier = active views stored in memory
account_id mod N
1/nth
of the members. So they use stale data to degrade gracefully.stateless tier = data persistence = Cassandra + Memcached
How to solve the problem? 3 Principles:
Client retries to ensure consistency.
Retry with idempotency and idempotency keys to allow clients to pass a unique value.
Retry with exponential backoff and random jitter. Be considerate of the thundering herd problem that servers that may be stuck in a degraded state and a burst of retries may further hurt the system.
For example, Stripe’s client retry calculates the delay like this…
def self.sleep_time(retry_count)
# Apply exponential backoff with initial_network_retry_delay on the
# number of attempts so far as inputs. Do not allow the number to exceed
# max_network_retry_delay.
sleep_seconds = [Stripe.initial_network_retry_delay * (2 ** (retry_count - 1)), Stripe.max_network_retry_delay].min
# Apply some jitter by randomizing the value in the range of (sleep_seconds
# / 2) to (sleep_seconds).
sleep_seconds = sleep_seconds * (0.5 * (1 + rand()))
# But never sleep less than the base sleep seconds.
sleep_seconds = [Stripe.initial_network_retry_delay, sleep_seconds].max
sleep_seconds
end
]]>Want an example? Go to see how Facebook scale its social graph data store.
]]>Why? Mentors cannot help with promotions but sponsors can.
A sponsor is
Video service over Http Live Streaming for mobile devices, which…
Server-side: In a typical configuration, a hardware encoder takes audio-video input, encodes it as H.264 video and AAC audio, and outputs it in an MPEG-2 Transport Stream
Client-side: client reads the index, then requests the listed media files in order and displays them without any pauses or gaps between segments.
For example, Apple Company will say…
Maximizing organization’s output.
A manager’s output = The output of his organization + The output of the neighboring organizations under his influence
This means that if a manager is not just a hierarchical supervisor but also a know-how manager (knowledge supplier), then he will have larger impact on both his own organization as well as neighboring organizations.
the definition of “manager” should be broadened: individual contributors who gather and disseminate know-how and information should also be seen as middle managers, because they exert great power within the organization.
Information gathering - the basis of all other managerial work
Information-giving
Decision-making, includes 2 kinds
==Nudging - advocating a preferred course of action, but you are not issuing a firm and detailed instruction. == it should be carefully distinguished from decision-making that results in firm, clear directives.
Being a role model. nothing leads as well as example. Values and behavioral norms are simply not transmitted easily by talk or memo, but are conveyed very effectively by doing and doing visibly.
By and large, none of the above can happen without a meeting. However, meeting is not an activity, it is an occasion or medium where activity happens.
Managerial Output = Output of organization = L1 × A1 + L2 × A2 +…
To maximize the output…
Leverage can be increased…
For example,
Positive leverage
Negative leverage
Shifting activities to those with higher leverage by DELEGATION
The “delegator” and “delegatee” must share a common information base and a common set of operational ideas or notions on how to go about solving problems
Being conscious
delegation without follow-through is abdication. how to monitor? QA
How to monitor the delegation of decision-making?
How Many Subordinates Should You Have?
Speeding up
time-management
For example
Principles
reducing Interruptions (the plague of managerial work) — how to solve? regularity and smoothing out workload.
All the discussions about flexi-time or dress-down Fridays or paternity leave only serve to mask the core issue, which is that certain job and career choices are fundamentally incompatible with being meaningfully engaged on a day-to-day basis with a young family.
governments and corporations aren’t going to solve this issue for us. If you don’t design your life, someone else will design it for you.
we have to be careful with the time frame that we choose upon which to judge our balance. A day is too short; “after I retire” is too long.
We need to approach balance in a balanced way. Lovely though physical exercise may be, there are other parts to life — there’s the intellectual side; there’s the emotional side; there’s the spiritual side. And to be balanced, I believe we have to attend to all of those areas.
There are four levels of energy.
What can we do in those states?
In the zombie state? Do not work or do little chore work.
In the robot state? Do preset work.
In the human state? Do exploratory work.
In the rockstar state? Do creative work.
Here are 30 “elements of value”.
Functional
Emotional
Life Changing
Social Impact
Specific knowledge
accountability
leverage
Alibaba Tmall Genie, Amazon Alexa, Amazon Ring
]]>Members of an organization give disproportionate weight to trivial issues.
Parkinson provides the example of a fictional committee whose job was to approve the plans for a nuclear power plant spending the majority of its time on discussions about relatively minor but easy-to-grasp issues, such as what materials to use for the staff bike shed, while neglecting the proposed design of the plant itself, which is far more important and a far more difficult and complex task.
Having a clear agenda of the meeting. Or even do not mix complex topics with easy ones.
]]>Reduce DNS lookups
<link rel="dns-prefetch" href="//www.example.com/" >
reuse TCP connections.
minimize number of HTTP redirects
use a CDN
eliminate unnecessary resources
Last-Modified
response header (not used often because nginx and microservices)compress assets during transfer
eliminate unnecessary request bytes
parallelize request and response processing
HTTP 1.x
HTTP 2.X
S - Single Responsibility Principle. A module should be responsible to one, and only one, actor. a module is just a cohesive set of functions and data structures.
O - Open/Closed Principle. A software artifact should be open for extension but closed for modification.
L - Liskov’s Substitution Principle. Simplify code with interface and implementation, generics, sub-classing, and duck-typing for inheritance.
I - Interface Segregation Principle. Segregate the monolithic interface into smaller ones to decouple modules.
D - Dependency Inversion Principle. The source code dependencies are inverted against the flow of control. most visible organizing principle in our architecture diagrams.
Structured programming is discipline imposed upon direct transfer of control.
OO programming is discipline imposed upon indirect transfer of control.
Functional programming: Immutability. is discipline imposed upon variable assignment.
Architecture is the shape of the software system. Thinking it as a big picture of physical buildings.
Together they serve a specific purpose, like a hospital is for curing patients and a school is for educating students.
Every software system provides two different values to the stakeholders: behavior and structure. Software developers are responsible for ensuring that both those values remain high.
::Software architects are, by virtue of their job description, more focused on the structure of the system than on its features and functions.::
Architecture serves the full lifecycle of the software system to make it easy to understand, develop, test, deploy, and operate. The goal is to minimize the human resources costs per business use-case.
]]>The most treasurable resources of a knowledge worker are willpower and power of attention. And truly restful breaks should recover these resources.
To recover willpower and power of attention, guiding principles are
Short-term
Mid-term
Long-term
What is the difference between debounce and throttle? try here
]]>“Progressive” means the improvement is not binary and terminal but evolutionary.
PWA = website optimized for mobile + manifest.json + service worker loading and registering
manifest.json is the easy part. Put the following into the example.com/manifest.json
{
"short_name": "Short",
"name": "Longer Name",
"icons": [
{
"src": "favicon.png",
"sizes": "192x192 150x150 144x144 64x64 32x32 24x24 16x16",
"type": "image/png"
}
],
"start_url": "/",
"display": "standalone",
"theme_color": "#de4c4f",
"background_color": "#f3f3f3"
}
And add the following into html <head>
<link rel="manifest" href="/manifest.json"/>
<link rel="apple-touch-icon" href="/favicon.png"/>
Then on both iOS and Android, users can add the site to the home screen.
Then … service worker loading and registering
The loading part I recommend create-react-app’s service worker loading script, which has good security practices and targets the cache first strategy. And it includes unregister as well.
The registering part is trickier - we added the following webpack plugin to prepare the service-worker.js.
// ...
plugins: [
// ...
new SWPrecacheWebpackPlugin(
{
mergeStaticsConfig: true,
dontCacheBustUrlsMatching: /\.\w{8}\./,
filename: 'service-worker.js',
minify: false,
navigateFallback: '/',
navigateFallbackWhitelist: [/^(?!\/__).*/],
staticFileGlobs: [
`${OUTPUT_DIR}/**`,
],
stripPrefix: OUTPUT_DIR,
staticFileGlobsIgnorePatterns: [/\.map$/, /asset-manifest\.json$/],
dynamicUrlToDependencies: {
'/index.html': glob.sync(path.resolve(`${OUTPUT_DIR}/**/*.js`)),
},
}
),
],
// ...
The tricky part here is that if you have SSR as we do - be careful to specify the dynamicUrlToDependencies; otherwise, cache may fail to be updated.
]]>Customers often provide feedbacks on MMRs and neutralizers. The product management team must take responsibility for reinforcing the startup’s differentiator. Once the market recognizes the startup’s advantage, every competitor will race to replicate it. The startup must invest in that differentiation to sustain their market lead.
]]>So there is no rollback plan and unnexpected scale problems occurred.
def get_user_by_names_or_ids(names=[], ids=[])
This causes memory leak - If you mutate those params, the mutations span across invocations. Accumulated data even blows up the memcache clusters.
]]>BASE (Availability over Consistency)
Although most NoSQL takes BASE priciples, they are learning from or moving toward ACID. e.g. Google Spanner provides strong consistency. MongoDB 4.0 adds support for multi-document ACID transactions.
]]>large dataset ⟶ scale out ⟶ data shard / partition ⟶ 1) routing for data access 2) replica for availability ⟶ consistency challenge
Any networked shared-data system can have only two of three desirable properties.
12 years later, the author Eric Brewer said “2 of 3” is mis-leading, because
Consequently, when there is no partition (nodes are connected correctly), which is often the case, we should have both AC. When there are partitions, deal them with 3 steps:
Pros of B tree
Pros of B+ tree
large dataset ⟶ scale out ⟶ data shard / partition ⟶ 1) routing for data access 2) replica for availability
The routing abstract model is essentially just two maps: 1) key-partition map 2) partition-machine map
hash and mod
Virtual buckets: key–(hash)–>vBucket, vBucket–(table lookup)–>servers
Consistent hashing and DHT
sort by primary key, shard by range of primary key
range-server lookup table (e.g. HBase .META. table) + local tree-based index (e.g. LSM, B+)
(+) search for a range (-) log(n)
Usercase: Yahoo PNUTS, Azure, Bigtable
]]>Moderate experience or less, or anyone who was not in a leadership or design position (either formal or informal) in their previous position
Describe one of your previous projects that was particularly interesting or memorable to you. Followup questions:
What about it made it interesting? What was the most challenging part of the project, and how did you address those challenges? What did you learn from the project, and what do you wish you had known before you started? What other designs/implementation methods did you consider? Why did you choose the one that you did? If you were to do the same project over again, what would you do differently?
Since the goal here is to assess the technical communication skill and interest level of someone who has not necessarily ever been in a role that they could conduct a crash course in, you should be prepared to keep asking them questions (either for more details, or about other aspects of the project). If they are a recent grad and did a thesis, that’s often a good choice to talk about. While this question is in many ways similar to the Resume Questions question from phone screen one, this question is intended to be approximately four times as long, and should get into proportionally more detail about what it is that they have done. As such, the scoring criteria are similar, but should be evaluated with both higher expectations and more data.
A great candidate will
Be able to talk for the full time about the project, with interaction from the interviewer being conversational rather than directing
Be knowledgeable about the project as a whole, rather than only their area of focus, and be able to articulate the intent and design of the project
Be passionate about whatever the project was, and able to describe the elements of the project that inspired that passion clearly
Be able to clearly explain what alternatives were considered, and why they chose the implementation strategy that they did.
Have reflected on and learned from their experiences
A good candidate will
May have some trouble talking for the full time, but will be able to with some help and questions from the interviewer
May lack some knowledge about the larger scope of the project, but still have strong knowledge of their particular area and pieces that directly interacted with them
May seem passionate, but be unable to clearly explain what inspired that passion
May be able to discuss alternatives to what they did, but not have considered them in depth
Have reflected on and learned from their experiences
A bad candidate will
Have difficulty talking for the full time. The interviewer may feel as if they are interrogating rather than conversing with the candidate
May lack detailed knowledge of the project, even within the area that they were working. They may not understand why their piece was designed the way it was, or may not understand how it interacted with other systems
Does not seem very interested in the project - remember that you are asking them about the most interesting project that they have done, they should be very - interested in whatever it was
May not be familiar with potential alternatives to their implementation method
Does not seem to have learned from or reflected on their experiences with the project. A key sign of this is that the answer to ‘what did you learn’ and ‘what would you do differently’ are short and/or nearly identical.
Is process a bad thing? My guiding principle or golden rule of process is to never ask for something from someone that does not directly help them to get their own work done.
When the tool overwhelms it isn’t used and so it doesn’t matter - e.g. a giant Gantt chart. The telemetry spreadsheet is more helpful.
This tool is the right level of complexity for projects from 10 to 5000 people in my experience.
benefits: it is a resources allocation guideline
What is a senior engineer? A team of senior engineers without junior engineers is a team of engineers
what’s the next step of a senior engineer?
how people write software, e.g. react/redux, npm. Here comes a model that affects all large JS apps - code splitting.
e.g. CSS is bad in code removability
avoid central configuration of your application at all cost
avoid central import problem: router imports component A, B, and C
avoid base bundle pile of trash
We have to become good at finding the right abstractions: Empathy and experience -> Right abstractions
]]>Andrew Chen believes rentention is king
]]>A team with the responsibility to measure, understand and improve the flow of users in and out of the product and business. Finance owns the flow of cash in and out of a company. Growth owns the flow of customers in and out of a product.
Sustainable Growth = multiplication of
e.g. amazon’s growth = multiplication of
More than just A/B tests:
Why does growth rate drops if no new optimization is made? It is because the prev optimization has converted its target cohort. There is always a limit and a S-curve in growth. People have to test out the potential new breakthroughs.
The larger the company, the larger the sample size, the less thoughtful the experiment can be. big companies like small optimizations
go deep in funnels and focus on critical points.