Post on 03-Feb-2023
Coming up
• Going distributed - When and Why?
• The landscape of Big Data systems - What are the apps?
When do we go distributed?
• A truly distributed design is usually a second/third generation solution
• Amazon started off as simple web application talking to a database 15+ years ago
• Twitter started out as a simple Ruby on Rails application talking to MySQL in 2006
When do we go distributed?
• As the application’s / organization’s complexity grows
• Data, request volume is too large for a single machine
• Your software needs to be deployed in multiple data centers
• Your teams deliver software in the form of services
Courtesy : Jeff Dean’s LADIS 2009 Keynote
Designing systems for scale
• Many production grade systems have been built and written about in recent times
• Need for a taxonomy that describes the big data systems landscape
A taxonomy for distributed systems
• Distributed Storage Systems
• Distributed Applications
• Monitoring & Management
• Personalization & Recommendation
Distributed Storage
• Distributed Filesystems
• Distributed/Parallel Databases
• Messaging and Notification engines
Distributed Filesystems
• Allows clients to access files from multiple networked hosts
• Clients don’t access underlying block storage directly, go through protocols
• Modern DFSs are good at providing replication & fault tolerance
Distributed Databases
• A database engine that allows storage and retrieval across different machines in a network, a.k.a NoSQL databases.
• Apache Hive, Amazon Dynamo, HadoopDB, FB Cassandra, Google Bigtable
• They tend to be non relational, distributed, open-source and horizontally scalable
• Are schema free, easy support for replication, eventually consistent (BASE over ACID)
Distributed Apps
• Data parallel programming frameworks
• Graph processing engines
• P2P content delivery
• Multi tenanted SaaS applications
• Content delivery networks
Monitoring and Management
• Distributed debuggers, tracers and profiling applications
• Monitoring systems
Personalization & Recommendation
• Recommendation engines
• Sentiment analyzers
• Personalized news & content discovery systems
</presentation>Visit
www.systemswemake.com
Follow on Twitter @systems_we_make