BigDataFR recommends: Zeta Architecture: Hexagon is the new circle
An enterprise architecture solution for scale and efficiency

« Data processing in the enterprise goes very swiftly from “good enough” to “we need to be faster!” as expectations grow. The Zeta Architecture is an enterprise architecture that enables simplified business processes and defines a scalable way for increasing the speed of integrating data into the business. Following a bit of history and a description of the architecture, I’ll use Google as an example and look at the way the company deploys technologies for Gmail.

Origin story and motivation

I’ve worked on a variety of different information systems over my career, each with their own classes of challenge. The most interesting from a capacity perspective was for a company that delivers digital advertising. The biggest technical problems in that industry flow from the sheer volume of transactions that occur on a daily basis. Traffic flows in all hours of the day, but there are certainly peak periods, which means all planning must revolve around the capacity during the peak hours. This solution space isn’t altogether different than that of Amazon; they had to build their infrastructure to handle massive loads of peak traffic. Both Amazon and digital advertising, incidentally, have a Black Friday spike.

Many different architectural ideas came to my mind while I was in digital advertising. Real-time performance tracking of the advertising platform was one such thing. This was well before real-time became a hot buzzword in the technology industry. There was a point in time where this digital advertising company was “satisfied” with, or perhaps tolerated, having a two-to-three-hour delay between making changes to the system and having complete insight into the effects of the changes. After nearly a year at this company, I was finally able to get a large architectural change made to streamline log collection and management. Before the implementation started, I told everyone involved what would happen. Although this approach would enable the business to see the performance within approximately 5-10 minutes of the time a change was made, that this would not be good enough after people got a feel for what real-time could deliver. Since people didn’t have that taste in their mouths, they wouldn’t yet support going straight to real-time for this information. The implementation of this architecture was in place a few months after I departed the company for a new opportunity. The implementation worked great, and after about three months of experience with the new architecture, my former colleagues contacted me and told me they were looking to re-architect the entire solution to go to real time.

You might be wondering why I have decided to share this anecdote. It’s simple: “Good enough” doesn’t exist when it comes to technology. It is just a matter of time before people want more and faster. I predicted that my colleagues’ expectations would quickly change, and it proved to be true.

All of those expectations got me thinking, “what could I do to protect against those new expectations that I knew were bound to happen?” Over the following few months, I worked through the ramifications to all the pieces of an enterprise architecture to help solve this. But there was still something missing. I couldn’t put my finger on it until I attended 2014 MesosCon in Chicago. Finally, the last piece of my architecture was sitting in front of me. I was missing global resource management. When I added that in, it opened up new doors and new ways to benefit from the rest of the pieces of my architecture. Suddenly it hit me: the strategic combination of all of these pieces was greater than the sum of each of the parts.
Read full article
By Jim Scott
Source: http://radar.oreilly.com