Lack of Smart Engineers Considered Harmful

Following the traditional outsourcing mantra, Google should be focusing on its core competencies, while outsourcing everything they possibly can of the base infrastructure to vendors.  Turns out that that the company actually does exactly the opposite.  A few weeks ago, Luiz André Barroso and Urs Hölzle of Google published a very interesting piece: The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines. Here is a quote [emphasis mine]:

Buy vs Build:

Traditional IT infrastructure makes heavy use of third-party software components such as databases and system management software, and concentrates on creating software that is specific to the particular business where it adds direct value to the product offering, for example, as business logic on top of application servers and database engines. Large-scale Internet services providers such as Google usually take a different approach in which both application-specific logic and much of the cluster-level infrastructure software is written in-house. Platform-level software does make use of third-party components, but these tend to be open-source code that can be modified inhouse as needed. As a result, more of the entire software stack is under the control of the service developer.

This approach adds significant software development and maintenance work but can provide important benefits in flexibility and cost efficiency. Flexibility is important when critical functionality or performance bugs must be addressed, allowing a quick turn-around time for bug fixes at all levels. It is also extremely advantageous when facing complex system problems because it provides several options for addressing them.

This is an interesting approach that works only if you have very talented individuals available to do design work. Infrastructure as a competitive advantage does not mean throwing money at the problem – if it were that simple, startups competing in fields with well-capitalized competitors would not have gotten off the ground. The entrenched incumbents would have crushed them by virtue of their superior capital resources.   To make infrastructure work for you, rather than be a drain on your finances and focus, you need the very best people who can turn the bits and pieces into a well tuned engine that enables you to do things that others simply cannot do. Joel Spolsky did some research around this problem, which he summarized on Joel on Software. His central thesis is summed up by the following quote:

The Creative Zen team could spend years refining their ugly iPod knockoffs and never produce as beautiful, satisfying, and elegant a player as the Apple iPod. And they’re not going to make a dent in Apple’s market share because the magical design talent is just not there. They don’t have it.

The mediocre talent just never hits the high notes that the top talent hits all the time.

As an engineer/technical manager with some of the worlds largest networks under my belt, I’ve repeatedly seen what Joel said  proven out in practice, and I will be naming names, because it helps to be specific.  At UUNET, I worked with people like Juzer, Najam, Bill Barns, Louie Mamakos, Parantap, Andrew Partan, Joe Malcolm, Mike O’Dell, Tim Smith et al.  When I was at AOL, I had the privilege of working with folks like Hung Le, Dr. Wu, Rich Colella, Mark Muehl, John Schanz, JR Mitchell, Girija, et al. At my current gig, I can’t even write down the full set of people before this becomes too long – folks like Warren, Jon, Bikash, Eiichi, Paul S, Paul G, Stephen S, Beck, Steve P and W, Sergei,  Nicolas G, Johnny J and the rest of the folks on the infrastructure and operations teams. At Level (3), a company I’ve never worked for, but whose engineering and architecture folks I am familiar with, there are engineers like Shane Amante, Nasser, Tozz, Scott Madley, Epperson, Dr. Gibbings etc. These engineers (I am naming a few representative examples), have repeatedly done things that the vast majority of people I am familiar with in the networking world – simply could not do. They have done work, which in design and execution would be beyond normal people. This is not an exhaustive list of networking talent – it would be the height of hubris to think that this is the entirety of the talent pool, but what are the chances of those level of individuals working on your network? Slim at best.  Best in the world infrastructure needs best in the world people. There is simply no way around that.  For companies that are in the telecom space, hiring second rate people will get you third rate networks.  If you are going to compete on basis of your infrastructure, you should be able to back that up with the appropriate people. To back it up with appropriate people, the executive management needs to read and understand what this is about. If they don’t, someone should send them a brief on Quark which Joel mentions in his post I quoted earlier.


10 Responses to Lack of Smart Engineers Considered Harmful

  1. GreatEvil says:

    Considering some of the maintenance bills I have seen come in for both hardware and software the savings in the support can easily off set some of the expense of having people write and maintain applications.

    Ever have Oracle show up and audit you? No matter how good you are, at fortune 100 scale you are going to have some book keeping errors and how many of them do you think will favor the vendor?

  2. Tony Cooper says:


    As always, I enjoy reading your posts. Furthermore, I can’t agree more with your assessment of the individuals you called out, many of whom I’ve had the privilege and absolute pleasure to work with myself.

    Your post got me thinking about what a “first rate engineer” is and how they got that way. It also seems to imply that having “first rate engineers” can fix anything…

    Missing from your post I think is the nod towards the role “good” management plays in this… it is not enough to have “first rate engineers” in the lineup. A business needs to have a chain of command that understands how to effectively take advantage of that talent to meet the business needs.

    To that end, I am a firm believer that most “first rate engineers” are the direct result of the strong leadership, guidance, support and often times, mentoring (in both direct and indirect form) their management chains provide(d). The product of this type of relationship are engineers that not only understand the tactical application of technology, but have also developed a keen business acumen that allows them to apply technology in a more strategic sense- ultimately resulting in an engineer who has the vision, talent, and salesmanship skills required to “engineer” a more efficient, practical, sustainable and most important, relevant infrastructure for the business.

    So just hiring a few “first rate engineers” is not enough; a business also needs “first rate management” that is actively and productively engaged.

    • vijaygill says:

      Tony, absolutely. The phrase ‘necessary but not sufficient’ comes to mind here. Without all cylinders firing, engineers by themselves can’t get you over the top, you need aligned organizations also populated with excellent folks. On the other hand, if you have a quasi-monopoly, or some other market distortion, then the above doesn’t hold.

  3. Lucky for you, Google seems to get it. 🙂

    From a 10-Q:

    To be successful, our network infrastructure has to perform well and be reliable. The greater the user traffic and the greater the complexity of our products and services, the more computing power we will need. We have spent and expect to continue to spend substantial amounts on the purchase and lease of data centers and equipment and the upgrade of our technology and network infrastructure to handle increased traffic on our web sites and to roll out new products and services. This expansion is expensive and complex and could result in inefficiencies or operational failures. If we do not expand successfully, or if we experience inefficiencies and operational failures, the quality of our products and services and our users’ experience could decline.


  4. […] can’t get there from here. This observation follows directly from my earlier article titled Lack of Smart Engineers Considered Harmful.  Here is a great quote about […]

  5. […] Lack of Smart Engineers Considered Harmful « Vijay Gill’s Blog Think about it! (tags: engineering shortage problem) […]

  6. Esteban Brenes says:

    Great articles Vijay, hope you post some more.

  7. […] issues above. Why?  Because the telecom companies simply do not have the talent in-house to do what it takes. Back in the glory days of ATT labs, when people like David Presotto, Rob Pike, Ken Thompson et al. […]

    • Alex says:

      Do they need to, we have a sort of a duopoly, quadopoly,monopoly in a way, funny how att was big into research and development as monopoly, but not anymore, we sort of have a duopoly with t-mobile and sprint being an underdog, though folks often regard their networks as being not as good though decent.

  8. Kane says:

    Hello Vijay,

    I love your blog and have been thinking alot about this specific post. I understand that education and mentors play a role but this are definitely not all the ingredients. As an engineer what do I need to have and how do I get to the kind of “state” that you would call smart?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: