Category Archives: amazon-dynamodb

Increase throughput in network load heavy topology storm with dynamo

I have a topology running on aws. I use M3 xlarge machines with 15GB ram, 8 supervisors. My topology is simple, I read from kafka spout -> [db o/p1] -> [db o/p2] -> [dynamo fetch] -> [dynamo write & kafka write] kafka

db o/ps are conditional. with latency around 100 - 150 ms.

But I have never been able to achieve a throughput of more than 300 msgs/sec.

What configuration changes are to be made so that I can get a throughput of more than 3k msgs/sec?

dynamo fetch bolt execute latency is around 150 - 220ms and dynamo read bolt execute latency is also around this number.

four bolts with parallelism 90 each and one spout with parallelism 30 (30 kafka partitions)

overall latency is greater than 4 secs.

topology.message.timeout.secs: 600 worker.childopts: "-Xmx5120m no. of worker ports per machine : 2

no of workers : 6 no of threads : 414 executor send buffer size 16384 executor receive buffer size 16384 transfer buffer size: 34 no of ackers: 24

org/apache/http/util/Args (java.lang.NoClassDefFoundError). Message payload is of type: String

My mule application is able to run with the following maven dependencies.

    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk</artifactId>
        <version>1.9.40</version>
    </dependency>

    <dependency> 
        <groupId>org.apache.httpcomponents</groupId> 
        <artifactId>httpcore</artifactId> 
        <version>4.4.1</version> 
    </dependency> 

    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.4.1</version>
    </dependency>

It runs smoothly, however as soon as I send POST to it, it generates the "NoClassDefFoundError"


ERROR 2015-06-11 12:34:10,406 [[dynamodb-01].connector.http.mule.default.receiver.02] org.mule.exception.DefaultMessagingExceptionStrategy: 
********************************************************************************
Message               : org/apache/http/util/Args (java.lang.NoClassDefFoundError). Message payload is of type: String
Code                  : MULE_ERROR--2
--------------------------------------------------------------------------------
Exception stack is:
1. org.apache.http.util.Args (java.lang.ClassNotFoundException)
  java.net.URLClassLoader$1:366 (null)
2. org/apache/http/util/Args (java.lang.NoClassDefFoundError)
  org.apache.http.conn.scheme.Scheme:90 (null)
3. org/apache/http/util/Args (java.lang.NoClassDefFoundError). Message payload is of type: String (org.mule.api.MessagingException)
  org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor:32 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.lang.ClassNotFoundException: org.apache.http.util.Args
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************

proxy_fcgi:error in server logs

I have android/iOS apps which are calling APIs. Our server is getting down once or twice a day. When it does it displays "Service unavailable" error. We are using zend APIs and DynamoDB if that makes any difference.

We are using bitnami server and created multiple virtual hosts on this. I tried to see logs Only susceptible thing I found is "proxy_fcgi:error". I tried to search but couldn't find exactly when we get this error.

Can please some one help me and identify or just tell me what could be the reason of server going down apart from traffic.