Several Solutions for Queue & Worker Systems Almost every dating site we’ve built has a job queue of some kind. We enqueue email sends, statistic updates, logging, fraud detection and more. Anything that doesn’t immediately impact the response we’re preparing for the user can be run in the background. Over the years we’ve tried several different queue/worker systems and we’d like to share our findings to help others decide on a technology. The solutions distinguish themselves in a few key areas: Scalability There are several separate scalability concerns to consider, such as the throughput of the queue, the number of the workers it can support and the maximum queue size. Priority Sending a password reset email shouldn’t wait until a batch of hundreds of thousands of “Your Daily Picks” emails have been processed. Administrative Tools Some solutions provide web based administration tools to check queue lengths, contents of the queue and to resubmit failed jobs. Scheduling Not all jobs should be executed immediately. Redundancy Beanstalk
Mapa de colaboración Prezi Examples — Eventlet v0.9.13.dev1 documentation Here are a bunch of small example programs that use Eventlet. All of these examples can be found in the examples directory of a source copy of Eventlet. Web Crawler examples/webcrawler.py #! WSGI Server examples/wsgi.py """This is a simple example of running a wsgi application with eventlet.For a more fully-featured server which supports multiple processes,multiple threads, and graceful code reloading, see: import eventletfrom eventlet import wsgi def hello_world(env, start_response): if env['PATH_INFO'] ! Echo Server examples/echoserver.py #! Socket Connect examples/connect.py """Spawn multiple workers and collect their results. Multi-User Chat Server examples/chat_server.py This is a little different from the echo server, in that it broadcasts the messages to all participants, not just the sender. Feed Scraper examples/feedscraper.py This example requires Feedparser to be installed or on the PYTHONPATH. Port Forwarder examples/forwarder.py Recursive Web Crawler
celery What’s a Task Queue? Task queues are used as a mechanism to distribute work across threads or machines. A task queue’s input is a unit of work, called a task, dedicated worker processes then constantly monitor the queue for new work to perform. Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task a client puts a message on the queue, the broker then delivers the message to a worker. A Celery system can consist of multiple workers and brokers, giving way to high availability and horizontal scaling. Celery is written in Python, but the protocol can be implemented in any language. Language interoperability can also be achieved by using webhooks in such a way that the client enqueues an URL to be requested by a worker. What do I need? Celery version 4.2 runs on, Python (2.7, 3.4, 3.5, 3.6)PyPy (5.8) This is the last version to support Python 2.7, and from the next version (Celery 5.x) Python 3.5 or newer is required. Get Started Bundles
Gone Fishin': Building Super Scalable Systems: Blade Runner Meets Autonomic Computing In The Ambient Cloud All in all this is still my favorite post and I still think it's an accurate vision of a future. Not everyone agrees, but I guess we'll see... "But it is not complicated. Contents: We have not yet begun to scale. Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. Tomorrow the numbers foreshadow a new Cambrian explosion of connectivity that will look as different as the image of a bare lifeless earth looks to us today. If you aren't Google, or a very few other companies, how can you possibly compete? What are we looking at here? Notice how global, how plentiful, and how fast the flashes flicker. Now imagine if all this power was available to your application. But even if you did have this power at your disposal, what could you do with it? We can only imagine what the scalable architectures of tomorrow will look like. The word "ambient" means "completely enveloping," like "ambient sound" or "ambient temperature." Radical?
Nice collection of icon sets for your Linux Desktop with installation instructions| Ubuntu PPA | Unixmen This is a nice collection of iconsets for your Linux Desktop, so if you are a fun of changing default themes of your Linux Distribution, then this post is for you. I tried to collect the nicest and most updated icons themes existing in this moment, i included also a detailed howto install the themes for Ubuntu/ LinuxMint and Fedora based distributions. I tested the icon themes in Ubuntu 12.04 Precise Pangolin Beta1, icons did work just fine. 1- Faience: 2- Meliae Iconset theme 3- Gnome Colors Ubuntu users, use the following PPA: sudo apt-get install ppa:gnome-colors-packagers/ppasudo apt-get update sudo apt-get install gnome-colors 4- Hydroxygen_iconset 5- Humanity Icons Colors Version For Ubuntu & LinuxMint use the following PPA: sudo add-apt-repository ppa:ravefinity-project/ppasudo apt-get updatesudo apt-get install humanity-colors 6- Vibrant Icons 7- Awoken 2.4 (PPA Ubuntu) 8- AdriX icon theme 9- Faenza Icons: PPA for Faenza icons for Ubuntu: 1- Download and extract the icons :
PushButton Engine - Modular Flash Game Development Greenhouse The German Energy Agency (Dena), in cooperation with the two German companies Bioltec Systems GmbH and Sunfarming GmbH, has set up a training and demonstration center for the German Energy Agency (DMA) for renewable energy, energy use of waste and sustainable mobility in the Caribbean combined with… In Kislovodsk, province of Kraj Stavropol, a 7 hectare greenhouse has been realized for the cultivation of lettuce and tomatoes. Director of Sun Valley, Mr. Roots has received a second sale and installation agreement from Dagan Agricultural Automation, its exclusive distribution partner in China. Category 5 Alquimi Renewables and its technology group Island AgTech, together with its strategic engineering partner Sprung Structures, based in Calgary, Alberta, have finalized development of the first fully certified Category 5 hurricane resistant greenhouse and blackhouse applications. Temperature and lighting conditions in greenhouses need to be as uniform as possible for crops to prosper.
The Perils of Asynchrony Every time you come across anything more than a rudimentary system that has some moderately serious performance needs, someone somewhere on the team considers using asynchronous processing to help reduce (perceived) response time. That person needs to be identified and quickly locked in a padded room . . . . Just kidding! Often that person is me and often I end up writing the code and relearning why doing things in an asynchronous way (that also meets a certain "near guarantee" SLA including DR and HA needs) is very very hard. So it was with some chagrin that I was tasked with coding some infrastructure components to implement a Task Queue for my current team. Fortunately this time I'm not writing financial systems software - there you typically need to guarantee that although a task is asynchronous that it will be done within some SLA (e.g. 2-3 seconds). Anyway the default solutions in Java for Asynchronous processing are 1) Threads (and java.util.concurrent - which is awesome) References