background preloader

The C10K problem

[Help save the best Linux news source on the web -- subscribe to Linux Weekly News!] It's time for web servers to handle ten thousand clients simultaneously, don't you think? After all, the web is a big place now. And computers are big, too. In 1999 one of the busiest ftp sites, cdrom.com, actually handled 10000 clients simultaneously through a Gigabit Ethernet pipe. And the thin client model of computing appears to be coming back in style -- this time with the server out on the Internet, serving thousands of clients. With that in mind, here are a few notes on how to configure operating systems and write code to support thousands of clients. Contents Related Sites See Nick Black's execellent Fast UNIX Servers page for a circa-2009 look at the situation. In October 2003, Felix von Leitner put together an excellent web page and presentation about network scalability, complete with benchmarks comparing various networking system calls and operating systems. Book to Read First I/O frameworks 1. 2. Related:  swabler

Design and Implementation of a High-performance TCP/IP Communications Library Download source code - 129 KB Introduction This article is the second of a multi-part series that will cover the architecture and implementation of components needed to create and maintain a robust, scalable, high performance, massive multiplayer online game server and game engine. The first article of the series focused on building a Scheduling Engine to drive organized, real-time change in a virtual world. This article focuses on the design and implementation of a TCP/IP communication component, designed to efficiently handle communications between the game server and the remote game clients (players). Background: BBS Games to MUDs Back in the 80's, I ran a modem-based bulletin board system (BBS) that let users dial in and leave messages for other users, share files, and play simple multi-player games. When I headed off for college in the early 90's, I had to shut down the BBS. Terminology - Sockets, and Outgoing / Incoming Connections Peer-to-Peer vs. Okay, enough confusion. Example Use

WebDAV Web Distributed Authoring and Versioning (WebDAV) is an extension of the Hypertext Transfer Protocol (HTTP) that allows clients to perform remote Web content authoring operations. A working group of the Internet Engineering Task Force (IETF) defined WebDAV in RFC 4918. The WebDAV working group concluded its work in March 2007, after the Internet Engineering Steering Group (IESG) accepted an incremental update to RFC 2518. Other extensions left unfinished at that time, such as the BIND method, have been finished by their individual authors, independent of the formal working group. Many modern operating systems provide built-in client-side support for WebDAV. History[edit] The W3C meeting decided to form an IETF working group, because the new effort would lead to extensions to HTTP, which the IETF had started to standardize. The protocol consists of a set of new methods and headers for use in HTTP. Implementations[edit] Servers[edit] For example: Clients[edit] Comparison of WebDAV software

A reusable, high performance, socket server class - Part 1 The following source was built using Visual Studio 6.0 SP5 and Visual Studio .NET. You need to have a version of the Microsoft Platform SDK installed Note that the debug builds of the code waste a lot of CPU cycles due to the the debug trace output. Overview Writing a high performance server that runs on Windows NT and uses sockets to communicate with the outside world isn't that hard once you dig through the API references. What does a socket server need to do? A socket server needs to be able to listen on a specific port, accept connections and read and write data from the socket. Before we can start accepting connections we need to have a socket to listen on. Hide Copy Code virtual SOCKET CreateListeningSocket( unsigned long address, unsigned short port); The server class provides a default implementation that's adequate in most circumstances. Note that we use a helper class, CSocket, to handle setting up our listening socket. Asynchronous IO Graceful shutdown Socket closure A simple server

How does a relational database work - Coding Geek When it comes to relational databases, I can’t help thinking that something is missing. They’re used everywhere. There are many different databases: from the small and useful SQLite to the powerful Teradata. But, there are only a few articles that explain how a database works. Are relational databases too old and too boring to be explained outside of university courses, research papers and books? As a developer, I HATE using something I don’t understand. Though the title of this article is explicit, the aim of this article is NOT to understand how to use a database. I’ll start with some computer science stuff like time complexity. Since it’s a long and technical article that involves many algorithms and data structures, take your time to read it. For the more knowledgeable of you, this article is more or less divided into 3 parts: A long time ago (in a galaxy far, far away….), developers had to know exactly the number of operations they were coding. O(1) vs O(n2) The concept Examples Merge

Google Architecture Update 2: Sorting 1 PB with MapReduce. PB is not peanut-butter-and-jelly misspelled. It's 1 petabyte or 1000 terabytes or 1,000,000 gigabytes. Google is the King of scalability. Information Sources Platform Linux A large diversity of languages: Python, Java, C++ What's Inside? The Stats Estimated 450,000 low-cost commodity servers in 2006 In 2005 Google indexed 8 billion web pages. The Stack Google visualizes their infrastructure as a three layer stack: Products: search, advertising, email, maps, video, chat, blogger Distributed Systems Infrastructure: GFS, MapReduce, and BigTable. Reliable Storage Mechanism with GFS (Google File System) Reliable scalable storage is a core need of any application. Do Something With the Data Using MapReduce Now that you have a good storage system, how do you do anything with so much data? Storing Structured Data in BigTable BigTable is a large scale, fault tolerant, self managing system that includes terabytes of memory and petabytes of storage. Hardware Misc

Linux Filesystem Hierarchy When migrating from another operating system such as Microsoft Windows to another; one thing that will profoundly affect the end user greatly will be the differences between the filesystems. What are filesystems? A filesystem is the methods and data structures that an operating system uses to keep track of files on a disk or partition; that is, the way the files are organized on the disk. The word is also used to refer to a partition or disk that is used to store the files or the type of the filesystem. Thus, one might say I have two filesystems meaning one has two partitions on which one stores files, or that one is using the extended filesystem, meaning the type of the filesystem. The difference between a disk or partition and the filesystem it contains is important. Before a partition or disk can be used as a filesystem, it needs to be initialized, and the bookkeeping data structures need to be written to the disk. Compliance requires that: Hardware Software /boot/boot.0300 /boot/boot.b

Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud "But it is not complicated. [There's] just a lot of it." -- Richard Feynman on how the immense variety of the world arises from simple rules. Contents: We have not yet begun to scale. Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. Tomorrow the numbers foreshadow a new Cambrian explosion of connectivity that will look as different as the image of a bare lifeless earth looks to us today. If you aren't Google, or a very few other companies, how can you possibly compete? What are we looking at here? Notice how global, how plentiful, and how fast the flashes flicker. Now imagine if all this power was available to your application. But even if you did have this power at your disposal, what could you do with it? We can only imagine what the scalable architectures of tomorrow will look like. The word "ambient" means "completely enveloping," like "ambient sound" or "ambient temperature." For a while I thought this might be true.

How Does Google do Planet-Scale Engineering for a Planet-Scale Infrastructure? How does Google keep all its services up and running? They almost never seem to fail. If you've ever wondered we get a wonderful peek behind the curtain in a talk given at GCP NEXT 2016 by Melissa Binde, Director, Storage SRE at Google: How Google Does Planet-Scale Engineering for Planet-Scale Infrastructure. Melissa's talk is short, but it's packed with wisdom and delivered in a no nonsense style that makes you think if your service is down Melissa is definitely the kind of person you want on the case. Oh, just what is SRE? what happens when a software engineer is tasked with what used to be called operations. Let that bounce around your head for awhile. Above and beyond all else one thing is clear: SREs are the custodian of production. Some of the highlights of the talk for me: The Destructive Incentives of Pitting Uptime vs Features. Other interesting topics in the talk are: How is SRE structured organizationally? Let's get on with it with it. The section title is my characterization.

How To Create Bootable USB Without Any Software In Windows 10 (Using Command Prompt) These days very few people burn ISO images on disks and use them as bootable media. Now, USB drives have replaced them as they are faster and a more convenient method to install an operating system on a machine. While there are many tools to make bootable USB drive, there’s one provided by even Microsoft, there’s a way to do the same without any software. All you need to do is fire up Command Prompt and follow some simple instructions. Let’s take a look at it: Insert your USB flash drive to your computer.Search for ‘cmd’ in Windows 10 from the taskbar, Command Prompt will appear. Most probably, you will see two Disk types. Disk 0 for hard drive and,Disk 1 for your USB Flash Drive with its total capacity. Also read: How To Monitor Network Usage In Windows 10 And Save Your Data? Since we are going to work on to create bootable USB drive, we will work with Disk 1. Type select disk 1 in the same windows and hit Enter. Copying files from an ISO image on your PC/DVD to USB manually:

The Best Place to Put Your Wi-Fi Router - Explained by Physics Jason Cole, a PhD scholar of Imperial College, London, has studied this plasma-electromagnetic waves’ interaction. He writes that the behavior of intense EM waves and plasma can be predicted using the Maxwell’s equations that date back to the 19th century. How Maths fixed the Wi-Fi in a flat? Jason used these equations to improve the Wi-Fi reception in his flat. In his model, he introduced a similar current and made it to oscillate; the resultant EM waves flow was dictated by Maxwell’s equations. The first lesson he learned is pretty much obvious: Wi-Fi router should be kept in line-of-sight to your device. Further, he found that sometimes signals stop changing and flicker without any particular reason at some places. So here we got our second lesson which is a bit more interesting and less obvious: if you are receiving a poor signal at some particular position, you should do a slight change in router’s position. Some more tips to fix-up your Wi-Fi by choosing the best place: 1. 2. 3. 4.

The Employment Status of Americans Visualized by Age Visualization Mobile Users: If you are having issues using the interactive version of this visualization, you can find a static version of it here. Data Notes A couple weeks ago, I published a graphic showing how marital status changes with age. I decided to do another one showing how employment status changes as well. This visualization uses Tableau to show the employment status for every age. C++Interview Questions and Answers What is C++? Released in 1985, C++ is an object-oriented programming language created by Bjarne Stroustrup. C++ maintains almost all aspects of the C language, while simplifying memory management and adding several features - including a new datatype known as a class (you will learn more about these later) - to allow object-oriented programming. C++ maintains the features of C which allowed for low-level memory access but also gives the programmer new tools to simplify memory management. C++ used for: C++ is a powerful general-purpose programming language. What is the difference between realloc() and free()? The free subroutine frees a block of memory previously allocated by the malloc subroutine. What is the difference between declaration and definition? The declaration tells the compiler that at some later point we plan to present the definition of this declaration. How do you write a function that can reverse a linked-list? for(; curnext! curnext->next = cur; } }

Related: