Category Archives: apache-httpcomponents

Async response streaming with Apache Async Http client

I am using apache async http client to stream objects from azure storage.

I only need to return the HttpResponse object which has the stream associated. My clients will actually have to read from that stream to store the file locally.

So Apache Async clients use a BasicAsyncResponseConsumer which actually buffers the entire file in local memory before calling the completed callback.

I am trying to create my own implementation of AbstractAsyncResponseConsumer so that I can stream the response body instead of actually storing it first but have been unsuccessful to do so till now.

Here is the bare bones cosumer class for reference ->

public class MyConsumer extends` AbstractAsyncResponseConsumer<HttpResponse> {
@Override
protected void onResponseReceived(HttpResponse response) throws HttpException, IOException {

}

@Override
protected void onContentReceived(ContentDecoder decoder, IOControl ioctrl) throws IOException {

}

@Override
protected void onEntityEnclosed(HttpEntity entity, ContentType contentType) throws IOException {

}

@Override
protected HttpResponse buildResult(HttpContext context) throws Exception {
    return null;
}

@Override
protected void releaseResources() {

}

}

And here is the code to send the request and return the response ->

public void getFile(HttpRequestBase request) {

    MyConsumer myConsumer = new MyConsumer();
    HttpAsyncRequestProducer producer = 
    HttpAsyncMethods.create(request);
    CompletableFuture<HttpResponse> future = new CompletableFuture<>();
    return Future<HttpResponse> responseFuture = 
    httpclient.execute(producer,myConsumer,                                                                                                                   
    new FutureCallback<HttpResponse>() {
      @Override
      public void completed(HttpResponse result) {
     //This is called only when all the response body has been read
     //future.complete(Result)

      }
      @Override                                                                      
      public void failed(Exception ex) {
      }
      @Override
      public void cancelled() {                                                                       
      }
   });

return future;

 }

I will be returning a CompletableFuture of the HttpResponse object to my clients.

They shouldnt be waiting for my http client to read all the response body first in local buffer.

They ideally should start copying directly from the stream provided in the response object.

What should I add inmy implementation of the consumer to get the desired result ?

How to stop camel http retry

We are using toD component of camel 2.18.1 to post a body to http web service but when the url is invalid camel is internally retrying for 2-3 minutes, which is something I do not want.

I explored further and found out that camel is internally using org.apache.commons.httpclient.HttpMethodDirector to perform the retry operation. How can stop Camel from doing this.

Part exception log is below.

<2017-03-29 16:33:59,084>:<>:<>:<>:<org.apache.commons.httpclient.HttpMethodDirector>:<INFO >:[Camel (camel-1) thread #20 - Threads]::<I/O exception (java.net.ConnectException) caught when processing request: Connection timed out (Connection timed out)>
<2017-03-29 16:33:59,085>:<>:<>:<>:<org.apache.commons.httpclient.HttpMethodDirector>:<INFO >:[Camel (camel-1) thread #20 - Threads]::<Retrying request>
<2017-03-29 16:36:06,313>:<>:<>:<>:<org.apache.commons.httpclient.HttpMethodDirector>:<INFO >:[Camel (camel-1) thread #20 - Threads]::<I/O exception (java.net.ConnectException) caught when processing request: Connection timed out (Connection timed out)>
<2017-03-29 16:36:06,314>:<>:<>:<>:<org.apache.commons.httpclient.HttpMethodDirector>:<INFO >:[Camel (camel-1) thread #20 - Threads]::<Retrying request>
<2017-03-29 16:38:13,545>:<>:<>:<>:<org.apache.commons.httpclient.HttpMethodDirector>:<INFO >:[Camel (camel-1) thread #20 - Threads]::<I/O exception (java.net.ConnectException) caught when processing request: Connection timed out (Connection timed out)>
<2017-03-29 16:38:13,546>:<>:<>:<>:<org.apache.commons.httpclient.HttpMethodDirector>:<INFO >:[Camel (camel-1) thread #20 - Threads]::<Retrying request>

here is the stack trace which I am getting

java.net.ConnectException: Connection timed out (Connection timed out)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at java.net.Socket.connect(Socket.java:538)
        at java.net.Socket.<init>(Socket.java:434)
        at java.net.Socket.<init>(Socket.java:286)
        at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
        at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
        at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
        at org.apache.camel.component.http.HttpProducer.executeMethod(HttpProducer.java:257)
        at org.apache.camel.component.http.HttpProducer.process(HttpProducer.java:152)
        at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
        at org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:124)
        at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:436)
        at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:119)
        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
        at org.apache.camel.processor.TryProcessor.process(TryProcessor.java:113)
        at org.apache.camel.processor.TryProcessor.process(TryProcessor.java:84)
        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
        at org.apache.camel.processor.Pipeline.access$100(Pipeline.java:44)
        at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:138)
        at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:88)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

finally here is the code being used in the route builder

 from("disruptor:" + "CallAPostService")
                    .routeId("CallAPostService")
                    .threads()
                    .log(LoggingLevel.INFO, LOG, "Processing id ${id} disruptor")
                    .doTry()
                        .process(new populateRequestBody()) // this processor sets the request xml to excahnge body
                                    .log(LoggingLevel.INFO, LOG, "Request XML = ${body}")
                                    .setHeader(Exchange.HTTP_METHOD, simple("POST"))
                                    .setHeader(Exchange.HTTP_BASE_URI, constant(""))
                                    .setHeader(Exchange.HTTP_PATH, constant(""))
                                    .setHeader(Exchange.HTTP_URI, simple(""))
                                                                    .setHeader(Exchange.HTTP

_URL, simple(""))
                                                                .toD("http://172.0.0.0:7001/myapp/cosumemessage" + "?bridgeEndpoint=true")                                                                
 .doCatch(Exception.class)                            
 .log(LoggingLevel.ERROR, LOG, "Processing id ${id} disruptor, Unknown Exception")
.doFinally();

Target host is not specified with apache component

I use spring boot I try to do a call to a web service.

My client localhost:9000 try to call localhost:8080

final RequestConfig config = RequestConfig.custom().setConnectTimeout(1000 * 1000).setConnectionRequestTimeout(1000 * 1000).setSocketTimeout(1000 * 1000).build();


final BasicCredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(new AuthScope("http://localhost", 8080, AuthScope.ANY_REALM), new UsernamePasswordCredentials("joe", "sixpack"));
final CloseableHttpClient client = HttpClientBuilder.create().setDefaultRequestConfig(config).setDefaultCredentialsProvider(credentialsProvider).build();

final ClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(client);
RestTemplate restTemplate2 = new RestTemplate(requestFactory);


ResponseEntity responseEntity = restTemplate2.patchForObject("/rest/members/{memberId}/card?cardId=", null, ResponseEntity.class, memberId, cardId);

When i do the call

org.apache.http.ProtocolException: Target host is not specified

Could someone please provide a simple Apache HttpComponents Client+Server example?

I have been looking for a way to implement network interaction in my java project and found out Apache HttpComponents is a solid choice. I have tried out a couple of examples from their tutorials an through out the net but still fail to construct the simplest localhost echoing server.

What I am specifically looking for is code for, say, Server.java and Client.java so that when I launch it, client sends a String to the server and receives an echoing response.

It would really help me to get things going.

Apache ConnectionClosedException: Connection closed,HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:344)

we have three applications A,B and C deployed on same tomcat server.There is an HTTP call that happens between A to B(REST CALL) and Another http call from B to C(REST CALL)

We were using synchronous HTTP call initially,recently we changed our code to Asynchronous call. We have used HttpAsyncClients for (Application A to call Application B) and for (Application B to call Application C)..

we are getting intermittent org.apache.http.ConnectionClosedException: Some time we are getting this exception in Logs of application A(Async Client to B) and some time in Application B(Async Client to C)

 CloseableHttpAsyncClient  client = HttpAsyncClients.custom()
                    .setMaxConnPerRoute(100))
                    .setMaxConnTotal(config.getInt(150)).build();

and for Every request we gave timeout of

final RequestConfig params = RequestConfig.custom().setConnectTimeout(300000) // 5min
                    .setSocketTimeout(300000).setConnectionRequestTimeout(300000).build();

Please find the stack trace below:

   org.apache.http.ConnectionClosedException: Connection closed
        at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:344) [httpcore-nio-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261) [httpcore-nio-4.4.5.jar:na]
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) [httpasyncclient-4.1.2.jar:na]
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) [httpasyncclient-4.1.2.jar:na]
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) [httpcore-nio-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) [httpcore-nio-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) [httpcore-nio-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) [httpcore-nio-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) [httpcore-nio-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [httpcore-nio-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) [httpcore-nio-4.4.5.jar:4.4.5]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]

Please Note this question is not duplicate of Intermittent ConnectionClosedException with httpasynclient stacj trace is completely different..And event this issue happens intermittent.

PoolingHttpClientConnectionManager – when removing stale connections the pool becomes empty

I am using org.apache.http 4.4.4

I am sending requests in high rate

In order to deal with stale connections i am closing all the stale connections whenever I see Exception in the finally block :

  try {
            //blocking
            response = apacheHttpSyncClient.execute(request);
        } catch (Exception e) {
//log
enter code here}
finally {
 request.releaseConnection();
                    cm.closeExpiredConnections();
}

when cm

 @Autowired
    PoolingHttpClientConnectionManager cm;

The exception I am getting for the x first calls is :

  java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:170)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)

afterwards I start getting this :

org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:286)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)

I suspect the the problem is that cm.closeExpiredConnections(); remove stale connection but does not create a new ones instead

Is there a way , interface which deal with steal connections and oonRemove , create new one?

How do I use the Apache HttpClient library to submit a PATCH request with JSON data?

I’m using Apache HTTP client v 4.3.4. How do I submit JSON data to a URL via the PATCH method? I have tried this

    // Create the httpclient
    HttpClient httpclient = HttpClientBuilder.create().build();

    // Prepare a request object
    HttpUriRequest req = null;
    if (method.equals(RequestMethod.PATCH))
    {
        req = new HttpPatch(url);
        req.setHeader("Content-type", "application/json");
        if (jsonData != null)
        {
            final StringEntity stringData = new StringEntity(jsonData.toString());
            req.setEntity(stringData);
        }   // if

but on the “req.setEntity” line, I get the compilation error, “The method is undefined”. Note that my request needs to send the JSON data as is, as opposed to putting it into a name-value param pair.

Apache Aync Client Not Shutting Down

I have successfully utilized Apache CloseableHttpAsyncClient to call 300,000+ web pages over a day and a half. These pages have a high latency rate and I need to perform other operations. Threading is a better option than Python multi-processing for me if I can get the pool to work. However, there seems to be a problem shutting the pool down. The only method I can find is .close(). I tried starting up and shutting down the client but after my program supposedly exits, an event loop is still running. I see no other way to call a shutdown. Below is my shutdown method. This is from my attempt without any threads being executed at all, just the thread managing the pool. I am using this code from Scala so I can more easily handle returned pages.

def close():PoolStatus.Value={
          try{
            asynClient.close()
          }catch{
            case t: Throwable => Logger.log(t.getMessage, levels.error)
          }
          PoolStatus.CLOSED
}

I receive no error just logging from the class.

17:58:58.021 [main] WARN  org.slf4j.Logger - Attempting to close and remove session
17:58:58.054 [main] INFO  org.slf4j.Logger - Crawl Complete
17:58:58.054 [main] INFO  org.slf4j.Logger - Terminating Scala

The program hangs after the last line in the main.

java.lang.NoSuchFieldError: INSTANCE (Selenium 2.53.0)

Here is another team's issue I am trying to resolve. I have tried multiple suggestions out there on the web to remove the httpclient/httpcore jar, lower its version and the rest of the fixes including clean .m2 repo but still running into below stacktrace. I have tried most of the routine fixes but still not luck. Any help would be greatly appreciated!!!

Stacktrace:

 java.lang.NoSuchFieldError: INSTANCE
 at org.apache.http.conn.ssl.SSLConnectionSocketFactory.(SSLConne
 ctionSocketFactory.java:144)
 at org.openqa.selenium.remote.internal.HttpClientFactory.getClientConnec
 tionManager(HttpClientFactory.java:71)
 at org.openqa.selenium.remote.internal.HttpClientFactory.(HttpClie
 ntFactory.java:57)
 at org.openqa.selenium.remote.internal.HttpClientFactory.(HttpClie
 ntFactory.java:60)
 at org.openqa.selenium.remote.internal.ApacheHttpClient$Factory.getDefau
 ltHttpClientFactory(ApacheHttpClient.java:252)
 at org.openqa.selenium.remote.internal.ApacheHttpClient$Factory.(A
 pacheHttpClient.java:229)
 at org.openqa.selenium.remote.HttpCommandExecutor.getDefaultClientFactor
 y(HttpCommandExecutor.java:96)
at org.openqa.selenium.remote.HttpCommandExecutor.(HttpCommandExec
utor.java:70)
at org.openqa.selenium.remote.HttpCommandExecutor.(HttpCommandExec
utor.java:58)
at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.st
art(NewProfileExtensionConnection.java:87)
at org.openqa.selenium.firefox.FirefoxDriver.startClient(FirefoxDriver.j
ava:271)
at org.openqa.selenium.remote.RemoteWebDriver.(RemoteWebDriver.jav
a:119)
at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:2
11)
at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:2
07)
at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:1

Dependency Tree

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ someProject ---
[INFO] com.someproject:jar:0.0.1-SNAPSHOT
[INFO] +- org.testng:testng:jar:6.9.9:compile
[INFO] | +- com.beust:jcommander:jar:1.48:compile
[INFO] | +- org.apache.ant:ant:jar:1.7.0:compile
[INFO] | | - org.apache.ant:ant-launcher:jar:1.7.0:compile
[INFO] | +- org.yaml:snakeyaml:jar:1.15:compile
[INFO] | - org.beanshell:bsh:jar:2.0b4:compile
[INFO] +- com.oracle:ojdbc7:jar:12.1.0.1:system
[INFO] +- apache-log4j:log4j:jar:1.2.15:system
[INFO] +- com.mercury.qualitycenter:otaclient:jar:9.2:system
[INFO] +- org.seleniumhq.selenium:selenium-java:jar:2.53.0:compile
[INFO] | +- org.seleniumhq.selenium:selenium-chrome-driver:jar:2.53.0:compile
[INFO] | +- org.seleniumhq.selenium:selenium-edge-driver:jar:2.53.0:compile
[INFO] | +- org.seleniumhq.selenium:selenium-ie-driver:jar:2.53.0:compile
[INFO] | | +- net.java.dev.jna:jna:jar:4.1.0:compile
[INFO] | | - net.java.dev.jna:jna-platform:jar:4.1.0:compile
[INFO] | +- org.seleniumhq.selenium:selenium-safari-driver:jar:2.53.0:compile
[INFO] | | - io.netty:netty:jar:3.5.7.Final:compile
[INFO] | +- org.seleniumhq.selenium:selenium-support:jar:2.53.0:compile
[INFO] | - org.seleniumhq.selenium:selenium-leg-rc:jar:2.53.0:compile
[INFO] +- org.seleniumhq.selenium:selenium-firefox-driver:jar:2.53.0:compile
[INFO] | +- org.seleniumhq.selenium:selenium-remote-driver:jar:2.53.0:compile
[INFO] | | +- cglib:cglib-nodep:jar:2.1_3:compile
[INFO] | | +- org.seleniumhq.selenium:selenium-api:jar:2.53.0:compile
[INFO] | | +- org.apache.httpcomponents:httpclient:jar:4.5.1:compile
[INFO] | | | +- org.apache.httpcomponents:httpcore:jar:4.4.3:compile
[INFO] | | | - commons-logging:commons-logging:jar:1.2:compile
[INFO] | | - com.google.guava:guava:jar:19.0:compile
[INFO] | +- commons-io:commons-io:jar:2.4:compile
[INFO] | - org.apache.commons:commons-exec:jar:1.3:compile
[INFO] +- org.apache.poi:poi:jar:3.10.1:compile
[INFO] | - commons-codec:commons-codec:jar:1.5:compile
[INFO] +- mysql:mysql-connector-java:jar:5.1.35:compile
[INFO] +- org.apache.poi:poi-ooxml:jar:3.10.1:compile
[INFO] | +- org.apache.poi:poi-ooxml-schemas:jar:3.10.1:compile
[INFO] | | - org.apache.xmlbeans:xmlbeans:jar:2.6.0:compile
[INFO] | | - stax:stax-api:jar:1.0.1:compile
[INFO] | - dom4j:dom4j:jar:1.6.1:compile
[INFO] | - xml-apis:xml-apis:jar:1.0.b2:compile
[INFO] +- com.google.code.gson:gson:jar:2.3.1:compile
[INFO] - junit:junit:jar:4.12:compile
[INFO] - org.hamcrest:hamcrest-core:jar:1.3:compile