Consulting-notes.com Website Review

providing senior consulting services for various open source software technologies specializing in Ja*a, Apache Camel, Apache ActiveMQ, Cas*andra, Elasticsearch, high volume messaging and various web application Over the past few years, I've been working with high volume real-time message based applications.  As we scale these systems out, resource usage becomes increasingly important and various trade-offs have to be considered. From a storage perspective, many of the NoSQL techs claim to handle mas* volumes of parallel requests and data with ease.  In the real processing.  As it turns out, this pattern is often a best practice (or even required) for reasonable performance from external systems I first encountered this use case when storing high volume messaging in Oracle.  The issue is that transaction commits are expensive and committing a single message at a time simply doesn't scale. Instead, we switched to aggregating messages in memory and then pas*ed these to Oracle via an array.  The net result was dramatically reduced load against the database, negligible delays in processing data and a theoretical increased chance of message loss...overall, this addressed our performance issues with relatively little impact. Then, as we moved into NoSQL solutions (Cas*andra, HDFS, ElasticSearch, etc), I revisited the need for batching...it was obvious that the same batching strategy still applied to these new techs as well.. Luckily, we are working with Apache Camel and its implementation of EIPs (Enterprise Integration Patterns) make solving these types of issues fairly straight forward.  In particular, the split and aggregator patterns are designed for just this type of message flow. For example, let's say I have 3 systems that process messages...A, B, C.  System A produces messages and sends them to system B.  system B does some processing and then sends them to system C for final processing. in Camel, this is expressed by the following simple route (as*uming ActiveMQ is used as a message bus in between systems)