The very first release that I took part in at PayPal was 9.0 which included a bunch of features and swapping out an SSL library. Secure Sockets Layer (SSL) is a standard security technology for establishing an encrypted link between two computers. This all seemed straightforward and, as per usual at the time, we shut off traffic to the site (can you believe we used to have to do that?) and began the release at 11pm PT on a Thu evening. It usually took us about 4 hours to push everything and by 3am the next morning we were ready to start taking traffic. Everything was fine until the east coast began to wake up and transfer money. At that point the entire site completely bogged down and we couldn’t process any transactions. Because our releases were six weeks worth of features, it was very difficult to diagnose the problem, hence one of the reasons for my strong belief in continuous deployment. We began “throwing hardware” at the problem by adding server capacity. This helped but definitely did not fix the problem. Being new and not being able to help much with debugging, I pitched in where I could. I was pretty sure I was going to be fired just weeks into the job. We broke into groups investigating different possible causes. Hours passed and we still struggled. Every time we thought we had it solved, traffic ramped back up, and transactions ground to a halt. Twenty-four hours passed and we still hadn’t solved it. Thirty-six hours passed and we kept debugging. Forty-eight hours later and we had an idea. It appeared that our servers were spending most of their time encrypting and decrypting messages between the three tiers (web-app-data). Further debugging revealed that the new SSL library was set to 1024-bit encryption where the old one was 256-bit, a big difference when it comes to the amount of computation cycles required. We reset this back to 256-bit early Sunday morning and everything went back to normal.
I wanted to share this story, not just because it has a lot of great lessons but also because it is an entertaining story. I am a huge fan of stories and storytelling. I think being a raconteur (a person who tells anecdotes in a skillful and amusing way) is a great leadership skill and an excellent way to impart information in a way that helps people retain it. Melanie Green, a communication professor at the University at Buffalo believes that a vivid, emotional story provides the extra push to make it important enough for most folks to remember. Researchers from Freie Universität Berlin, reported that when we read or hear stories we begin predicting and imagining what drives individuals in the story. This helps us see situations from different perspectives. There could even be benefits to strengthening relationships and bonding from stories. Researchers from Princeton University have shown that when you listen to a story, your brain waves begin to synchronize with those of the storyteller.
In a paper in the Proceedings of the National Academy of Sciences in 2014 titled Embers of society: Firelight talk among the Ju/’hoansi Bushmen the author Polly Wiessner, a cultural anthropologist, stated that, “stories told over firelight helped human culture and thought evolve by reinforcing social traditions, promoting harmony, and equality, and sparking the imagination to envision a broad sense of community.” In her study, Wiessner found that during the day, only 6% of the talk between the people in the San communities consisted of stories. But, at night by the campfire, 81% of the talk are stories. Weissner said, “Stories expanded the virtual social universe by reawakening feelings.”
Stories take all shapes and forms. One of the shortest stories ever written is Ernest Hemingway’s tear jerking "For sale: Baby shoes. Never worn." The longest story might go to Marcel Proust’s In Search of Lost Time, divided into seven volumes and containing over 1.5 million words. While there are unlimited plots, the American writer and satirist, Kurt Vonnegut, humorously outlines that there are only a few narrative arcs that keep getting repeated. As a leader using storytelling as a tool, you don’t need much to make a compelling story - a character, a challenge, and a resolution. When you want to bring people together, make information stick, pass along a lesson, or inspire people, instead of just talking to your team, try telling them a story.
There's a story by Augusto Monterroso that reads: "When they woke up, the dinosaur was still there." The power of just a few words.
typical alpha risk vs beta risk. we focsued too much on the alpha for the troubleshooting, and it was contribtued by beta overall