Bdtjtk

I and my team is stuck in a very silly case of HttpClientErrorException let me first give overview of my work scenario.

I have a micro-service stack with following things:

• Mysql - database


• keycloak - user management


• Eureka - discovery


• Zuul - Reverse Proxy


• Zipkin and Sleuth- tracers


• Microservice Hotel - everything related to the Hotel Entity(record of hotels, CRUD for hotels, and User accounts related to them).


• Microservice Room - all about rooms


• Microservice Price - all about prices


• Microservice Scheduler - A microservice which keeps track of data updates and Syncs it with another system(third party).

In scheduler I am using spring Scheduler with cron

@Scheduled(cron = "0 0/5 * * * ?")

public void prepareDataForSync() {...

The purpose of code in prepareDataForSync is to obtain data about each hotel and check current state and if any change is deducted pass it on to third party. Now here comes the real problem:

I make a rest service call to obtain list of hotels from my scheduler:

    @Scheduled(cron = "0 0/5 * * * ?")

public void prepareDataForSync() {



    Set<Long> unChangeableAllc = new HashSet<Long>();



    List<MessageCenter> messages = new ArrayList<MessageCenter>();

    String hotelURL = "http://hoteldata/hotel/allActive";



    try {//loop over hotel data and process further...

All these services are running in docker environment with each service having own container and communicating via docker networking.

Now when I start the services including Hotel and Scheduler, everything work fine for few hours but then I get following exception in my logs and service is no more syncing with third party.

org.springframework.web.client.HttpClientErrorException: 400 null

at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:63)

at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)

at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653)

at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)

at org.springframework.web.client.RestTemplate.getForEntity(RestTemplate.java:312)

at com.channelmanager.allocationservices.controllers.RoomAllocationsController.prepareDataForSync(RoomAllocationsController.java:1159)

at sun.reflect.GeneratedMethodAccessor257.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)

at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)

at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

At line 1159 I have a service call only:

ResponseEntity<String> hotelResponse = restTemplate.getForEntity(hotelURL, String.class);

I checked logs in corresponding Hotel service, logs there displays the service request was received and data was collected and written to response stream but as the exception shows I never received any response and get this Exception.

Logs from Hotel service:

2018-12-31 06:06:00 [http-nio-9501-exec-5] DEBUG o.s.web.servlet.DispatcherServlet - DispatcherServlet with name 'dispatcherServlet' processing GET request for [/public/hotel]

2018-12-31 06:06:00 [http-nio-9501-exec-5] DEBUG o.s.w.s.m.m.a.RequestMappingHandlerMapping - Looking up handler method for path /public/hotel

2018-12-31 06:06:00 [http-nio-9501-exec-5] DEBUG o.s.w.s.m.m.a.RequestMappingHandlerMapping - Returning handler method [public org.springframework.http.ResponseEntity<java.util.List<com.channelmanager.hoteldata.models.Hotel>> com.channelmanager.hoteldata.controllers.HotelUserPublicController.getAllHotel()]

2018-12-31 06:06:00 [http-nio-9501-exec-5] DEBUG o.s.web.servlet.DispatcherServlet - Last-Modified value for [/public/hotel] is: -1

2018-12-31 06:06:00 [http-nio-9501-exec-5] DEBUG o.s.w.s.m.m.a.HttpEntityMethodProcessor - Written [[Hotel [id=1, name=testhotel, createdBy=cmadmin, modifiedOn=2018-04-27T13:54:43, createdOn=2018-04-27T13:54:43, enabled=true, ...]]] as "application/json" using [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@a50d709]

2018-12-31 06:06:00 [http-nio-9501-exec-5] DEBUG o.s.web.servlet.DispatcherServlet - Null ModelAndView returned to DispatcherServlet with name 'dispatcherServlet': assuming HandlerAdapter completed request handling

2018-12-31 06:06:00 [http-nio-9501-exec-5] DEBUG o.s.web.servlet.DispatcherServlet - Successfully completed request

If I restart the Scheduler service it starts working again, but again in few hours i have same issue. As a workaround currently I have set up a cron on the server to restart the service every 2 hours but it is really a bad workaround, I can't rely on this in the production and need to get to the root of problem.

I have have googled and tried to go through any HttpClientErrorException based question, but nothing made sense to me.

Please let me know if more information is required from my end.

EDIT:

Docker Stats Output:

0b7d20c5a566 schedular 0.12% 1.365GiB / 31.41GiB 4.34% 0B / 0B 3.27MB / 0B 64

TOP Output inside container

The error-code implies an error on the other end - 400-errors are not located on your end. Have you tried dumping the response (including headers)? Also, did you try to re-authenticate, perhaps reset cookings, etc? Did you contact the other end? How did they respond to it?