13

Velocity 2011 - Big Data, Scale Dirty

 4 years ago
source link: https://www.otsukare.info/2011/06/14/velocity-2011-big-data-scale-dirty
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

Velocity 2011 - Big Data, Scale Dirty

Mar 14 juin 2011by Karl Dubost (Worked at Opera from 2010 to 2013)

BigData / NoSQL Operations Workshop: How to Scale Dirty and Influence People

Chef is a powerful tool for maintaining and describing the software and configurations that let a machine provide its services.

The more we can decouple, the more we can scale.

They define Chef as a make for machines and as Git for machines. If the code is a 100 lines range, it is very easy to be maintenable. Smaller codes help a lot. Chef helps describe a good high level mental model for your machine. Even if you do not know Ruby, you can read the recipes for configuring the machine.

Code is inventory, the more lines of code, the harder it will be to move away from them.

A written code is legacy. The bigger the code, the closer from the Web it is. Plenty of legacy constraints.

Product’s job: give me, Engineering says: No way\^W\^W Yes!

It reminds me of long discussions between developers and product owners. The issue is always the same and comes from a lack of dialog and understanding of the consequences of decision. People should have clear KPI to be able to take decisions.

Do not try to solve problems you would like to have.

Or basically the old “If it is not broken, do not fix it.”

The two speakers are introducing Resque

“Resque (pronounced like “rescue”) is a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later.”

They also used Flume for log aggregation.

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.

do not write unit tests for High perf sites. Real world will always bites you. Watch matrix and spend time on recovery techs

Typically you can spend a lot of time writing tests that will never fail, but you are loosing time. It is better to be reactive and monitoring metrics and improve your techniques of recovery. How the code is designed to be tolerant for failures. Basically how is your code planned for failures?

Even if the code smells funny inside, but is super effective, then it is beautiful code

The important is to have code which is resistant to performance issues. Software purity doesn’t achieve that much in face of real world.

do not spend time in meetings, write code.

Engineers spending too much time to try solving issues in a meeting room instead of writing code and testing the results live is wasted time. Play with the real world data.


report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK