Backstage Blog - Roshi: a CRDT system for timestamped events - SoundCloud Developers. Let's talk about the stream. The SoundCloud stream represents stuff that's relevant to you primarily via your social graph, arranged in time order, newest-first. The atom of that data model, an event, is a simple enough thing. TimestampUser who did the thingIdentifier of the thing that was done For example, If you followed A-Trak, you'd want to see that repost event in your stream. Easy. Data models Fan out on write means everybody gets an inbox. That's how it works today: we use Cassandra, and give each user a row in a column family. At some point, those caveats and restrictions started affecting our ability to iterate on the stream. The alternative is fan in on read. When A-Trak reposts Skrillex, it's a single append to A-Trak's outbox.
Of course, reads are difficult. Yet we saw potential here. Enter the CRDT If you're into so-called AP data systems, you've probably run into the term CRDT recently. Roshi The result of our work is Roshi, a distributed storage system for time-series events. How imgix Built A Stack To Serve 100,000 Images Per Second. By Kelly Sutton, Chief Product Officer at Imgix. What We Do With over 60% of the average webpage’s weight being image content, serving the best image in the smallest payload is an increasingly critical concern for both businesses and developers. Every additional second of load time for a page will increase its bounce rate by 7%, according to KISSMetrics.
Imagine losing 7% of the revenue from an e-commerce website simply because the images were suboptimal! Imgix is a real-time image processing and delivery service that works with your existing images. It was designed from the ground up to empower businesses and developers to serve the most optimal images under any circumstance. Individual customers have been able to eliminate millions of images from storage and generate an infinite number of derivative images on the fly. The Technical Challenge imgix has more than 80 different URL parameters that can be layered and combined for sophisticated effects.
Engineering at imgix The Early Versions. High Scalability - Update: Instagram Improved Their App's Performance. Here's How. When you find your mobile application that ran fine in the US is slow in other countries, how do you fix it? That’s a problem Facebook talks about in a couple of enlightening videos from the @scale conference. Since mobile is eating the world, this is the sort of thing you need to consider with your own apps. In the US we may complain about our mobile networks, but that’s more #firstworldproblems talk than reality. Mobile networks in other countries can be much slower and cost a lot more. Facebook found in the US there’s 70.6% 3G penetration with 280ms average latency. Chris also talked about Facebook’s comprehensive research on who uses Facebook and what kind of phones they use.
It turns out the typical phone used by Facebook users is from circa 2011, dual core, with less than 1GB of RAM. Facebook has moved to a product organization. Each team is responsible for the end-to-end performance and reliability for their product. Turning the database inside-out with Apache Samza | Confluent. This is an edited and expanded transcript of a talk I gave at Strange Loop 2014. The video recording (embedded below) has been watched over 8,000 times. For those of you who prefer reading, I thought it would be worth writing down the talk. Databases are global, shared, mutable state. That’s the way it has been since the 1960s, and no amount of NoSQL has changed that. However, most self-respecting developers have got rid of mutable global variables in their code long ago. So why do we tolerate databases as they are? A more promising model, used in some systems, is to think of a database as an always-growing collection of immutable facts.
This talk introduces Apache Samza, a distributed stream processing framework developed at LinkedIn. At its core is a distributed, durable commit log, implemented by Apache Kafka. What do we have to gain from turning the database inside out? This talk is about database architecture and application architecture. Let’s talk about databases. Rules - Auth0 Docs. Rules are code snippets written in JavaScript that are executed as part of the authentication pipeline in Auth0. This happens every time a user authenticates to an application. Rules enable very powerful customizations and extensions to be easily added to Auth0. An App initiates an authentication request to Auth0 (Step 1), Auth0 routes the request to an Identity Provider through a configured connection (Step 2).
The user authenticates successfuly (Step3), the user object that represents the logged in user is passed through the rules pipeline and returned to the app (Step 4). Here are a few examples. Auth0 Rules are implemented in JavaScript. Notice you can chain rules together to keep functionality modular and easy to understand. The simplest example: a Hello World This rule will add a hello attribute to all users authenticating through any provider. function (user, context, callback) { user.hello = 'world'; console.log('===> set "hello" for ' + user.name); callback(null, user, context)} Starbucks Does Not Use Two-Phase Commit - Enterprise Integration Patterns.
Hotto Cocoa o Kudasai I just returned from a 2 week trip to Japan. One of the more familiar sights was the ridiculous number of Starbucks (スターバックス) coffee shops, especially around Shinjuku and Roppongi. While waiting for my "Hotto Cocoa" I started to think about how Starbucks processes drink orders. Starbucks, like most other businesses is primarily interested in maximizing throughput of orders. Correlation By taking advantage of an asynchronous approach Starbucks also has to deal with the same challenges that asynchrony inherently brings. Exception Handling Exception handling in asynchronous messaging scenarios can be difficult. Write-off - This error handling strategy is the simplest of all: do nothing.
All of these strategies are different than a two-phase commit that relies on separate prepare and execute steps. Conversations The coffee shop interaction is also a good example of a simple, but common Conversation pattern. In summary we can see that the real world is often asynchronous. BoundedContext. Team organization · requirements analysis · application integration · domain driven design tags: Bounded Context is a central pattern in Domain-Driven Design. It is the focus of DDD's strategic design section which is all about dealing with large models and teams. DDD deals with large models by dividing them into different Bounded Contexts and being explicit about their interrelationships. DDD is about designing software based on models of the underlying domain. A model acts as a UbiquitousLanguage to help communication between software developers and domain experts. It also acts as the conceptual foundation for the design of the software itself - how it's broken down into objects or functions.
As you try to model a larger domain, it gets progressively harder to build a single unified model. Bounded Contexts have both unrelated concepts (such as a support ticket only existing in a customer support context) but also share concepts (such as products and customers). . . Microservices. "Microservices" - yet another new term on the crowded streets of software architecture. Although our natural inclination is to pass such things by with a contemptuous glance, this bit of terminology describes a style of software systems that we are finding more and more appealing.
We've seen many projects use this style in the last few years, and results so far have been positive, so much so that for many of our colleagues this is becoming the default style for building enterprise applications. Sadly, however, there's not much information that outlines what the microservice style is and how to do it. In short, the microservice architectural style [1] is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.
These services are built around business capabilities and independently deployable by fully automated deployment machinery. Componentization via Services. How many formats do I need for HTML5 video? « Zencoder Cloud Encoding Blog. EDIT: We’ve posted an updated guide on HTML5 video formats. Go check it out! The <video> tag in HTML5 is a great thing. It enables native video playback in all current browsers, rather than relying on a plugin like Flash. It opens the door to web video on devices that don’t support Flash. And it enables codecs that previously weren’t web playable. But if you’re publishing video content, this can be confusing. To really support web video, how many output versions do you need to create? There isn’t a single answer to this question, so here is a short guide to picking outputs for HTML5 video. What web formats? There are three video formats that work natively in some browsers. Minimally, you must use MP4 + H.264, with AAC or MP3.
Beyond that, use either WebM + VP8 or Ogg + Theora with Vorbis audio for other browsers. If you only pick one or the other, which one should you use – Ogg or WebM? In our opinion, WebM is the future of open video. What mobile devices? 1. 2. 3. What about streaming? 1. The Engine Room - TrackMaven. TrackMaven has begun hosting a Monthly Challenge meetup! Each month, we will name a general topic, a new technology, or something in between. We'll collect a few resources and examples to get everyone started (hence this post), then we'll meet up in a month to share short presentations on everyone's new projects. Our first topic is Elasticsearch, an incredibly powerful search and analytics engine. Go here for a high level, buzzword-heavy overview, or just jump into the documentation if you're feeling bold. Built on top of Lucene, Elasticsearch is most frequently used to add full text search functionality; it comes out of the box with a rich query language that supports fuzzy matching and advanced parsing patterns.
We'll go into the details of a sample project to get you started below. Elasticsearch communicates over a RESTful API using JSON. Let's get started! /bin/elasticsearch By default, Elasticsearch sits on port 9200. In your browser and see something like: Now let's put some data in! Как Docker помог нам достичь (почти) невозможного / Хабрахабр. С тех пор как мы начали работать над Iron.io, мы пытались решить проблему поддержания наших IronWorker-контейнеров в актуальном состоянии относительно новых сред выполнения и пакетов Linux.
В течение последних двух лет IronWorker использовал одну и ту же среду выполнения без изменений. Пока, несколько недель назад, мы не выпустили в продакшен различные окружения для языков программирования. С момента создания нашего сервиса, мы использовали только один контейнер, который содержал набор языковых сред и бинарных пакетов — Ruby, Python, PHP, Java, .NET и другие языки, а также библиотеки такие как ImageMagick, SoX и другие. Этот контейнер и стратегия его использования начали устаревать, равно как и Ruby 1.9.1, Node 0,8, Mono 2 и прочие языки со старыми версиями, которые использовались в стеке по умолчанию. Со временем проблема стала ещё острее, поскольку люди начали использовать новые вещи, но были вынуждены изменять свой код для работы со старыми версиями языков. … и тогда пришел Docker. Implementing Blue-Green Deployments with AWS. An important technique for reducing the risk of deployments is known as Blue-Green Deployments.
If we call the current live production environment “blue”, the technique consists of bringing up a parallel “green” environment with the new version of the software and once everything is tested and ready to go live, you simply switch all user traffic to the “green” environment, leaving the “blue” environment idle. When deploying to the cloud, it is common to then discard the idle environment if there is no need for rollbacks, especially when using immutable servers. If you are using Amazon Web Services (AWS) as your cloud provider, there are a few options to implement blue-green deployments depending on your system’s architecture. Since this technique relies on performing a single switch from “blue” to “green”, your choice will depend on how you are serving content in your infrastructure’s front-end.
Single EC2 instance with Elastic IP Multiple EC2 instances behind an ELB Conclusion. SOA Series Part 3: Documenting and Generating your APIs - LivingSocial's Technology Blog. This is the third in a series of seven posts on service-oriented architecture derived from a workshop conducted by Cloves Carneiro and Tim Schmelmer at Abril Pro Ruby. The series is called SOA from Day One – A love story in 7 parts. APIs will only be used when there is documentation on how to use them. In part three of this series of articles we recommend you think about how to structure and document your APIs from day one. Standards and Consistency To freely quote Andrew S. Tannenbaum: “The Good Thing About Standards: there are so many of them” Even if you say your API is RESTful, it doesn’t really say much in detail.
Heroku recently open-sourced its own HTTP API Design Guidelines to answer some of these questions for their use cases, and they are inviting everyone in the community to contribute. Specifying your Endpoints “What do you mean, I need to specify my application’s endpoints … can’t you read the routes file?!” No, that is unfortunately just not good enough, sorry! Ruby Tools. Demos using AWS with Node.JS and an AngularJS frontend - RecursiveRobot | E. John Feig. I recently decided to build some reusable code for a bunch of projects that I’ve got queued up. I wanted some backend components that leveraged a few of the highly scalable Amazon AWS services. This ended up taking me a month to finish, which is way longer than the week that I had intended to spend on it. (It was a month of nights and weekends, not as much free time as I’d hoped for in January.) Oh, and before I forget, here’s the GitHub repo.
This project’s goal is to build small demo utilities that should be a reasonable approximation of what we might see in an application that uses the aws-sdk node.js module. Here’s a temporary EBS instance running to demonstrate this. ToC Data Set Both DynamoDB and RDS are going to use the same basic data set. The same schema will be used for both Dynamo and RDS, almost.
DynamoDB Using the above schema, we set up a couple Dynamo tables. There are just a couple of things going on in the DynamoDB demo. Initialize Get all the users Insert or update a user <? The Four Architectures that will inspire your programming. This is the Big 4. The architectures, that may help your work. They may influence the way you think about programming. 1. The Hexagonal Architecture (also known as the Ports and Adapters) Originally described by Alistair Cockburn in Enhanced and popularised by Steve Freeman and Nat Pryce, thanks to their book: Clean Architecture Originally described by Ivar Jacobson, but popularised by Uncle Bob: Two separate concepts, but together create quite a unique architecture. James Coplien, Trygve Reenskaug Desribed in James' book: and here: What I think about them I don't treat them as competitors. The most mind-blowing architecture is definitely DCI. The most practical architecture is possibly DDD/CQRS.
The DCI Architecture: A New Vision of Object-Oriented Programming. The Clean Architecture. Hexagonal architecture.