Bug description Getting Error: getting embedding channel not registered to an event loop when calling model.embed (azure open ai) I have tried all kinds of things with no luck, like changing the netty version, Spring AI version with no luck
Environment Spring AI V1.0.0 , got the error in mac as well in linux
Steps to reproduce calling the model.embed(text) method
Expected behavior no errors, just a float[]
java.lang.IllegalStateException: channel not registered to an event loop
at io.netty.channel.AbstractChannel.eventLoop(AbstractChannel.java:163)
at com.azure.core.http.netty.implementation.NettyUtility.closeConnection(NettyUtility.java:79)
at com.azure.core.http.netty.implementation.NettyAsyncHttpResponse.close(NettyAsyncHttpResponse.java:116)
at com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:249)
at com.azure.core.http.policy.RetryPolicy.processSync(RetryPolicy.java:161)
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
at com.azure.core.http.policy.AddHeadersFromContextPolicy.processSync(AddHeadersFromContextPolicy.java:67)
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
at com.azure.core.http.policy.RequestIdPolicy.processSync(RequestIdPolicy.java:77)
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
at com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)
at com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
at com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)
at com.azure.core.implementation.http.rest.SyncRestProxy.send(SyncRestProxy.java:62)
at com.azure.core.implementation.http.rest.SyncRestProxy.invoke(SyncRestProxy.java:83)
at com.azure.core.implementation.http.rest.RestProxyBase.invoke(RestProxyBase.java:124)
at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:95)
at jdk.proxy2/jdk.proxy2.$Proxy128.getEmbeddingsSync(Unknown Source)
at com.azure.ai.openai.implementation.OpenAIClientImpl.getEmbeddingsWithResponse(OpenAIClientImpl.java:2361)
at com.azure.ai.openai.OpenAIClient.getEmbeddingsWithResponse(OpenAIClient.java:168)
at com.azure.ai.openai.OpenAIClient.getEmbeddings(OpenAIClient.java:620)
at org.springframework.ai.azure.openai.AzureOpenAiEmbeddingModel.lambda$call$1(AzureOpenAiEmbeddingModel.java:143)
at io.micrometer.observation.Observation.observe(Observation.java:565)
at org.springframework.ai.azure.openai.AzureOpenAiEmbeddingModel.call(AzureOpenAiEmbeddingModel.java:142)
at org.springframework.ai.embedding.EmbeddingModel.embed(EmbeddingModel.java:67)
at org.springframework.ai.embedding.EmbeddingModel.embed(EmbeddingModel.java:49)
at com.adc.rag.pg.repo.PostgresDAO.getEmbeddings(PostgresDAO.java:86)
at com.adc.rag.pg.repo.PostgresDAO.saveADCVectorDocument(PostgresDAO.java:130)
at com.adc.rag.pg.repo.PostgresDAO.saveVectorDocuments(PostgresDAO.java:235)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:355)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:769)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:379)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:769)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:721)
at com.adc.rag.pg.repo.PostgresDAO$$SpringCGLIB$$0.saveVectorDocuments(
Comment From: dev-jonghoonpark
I have tested it but did not find any issues. Please provide reproducible code.
Comment From: jego026
This error occurs intermittently—specifically, it tends to appear when we call the embeddings API multiple times in quick succession while processing a large batch of documents. Notably, the error does not occur when processing just one or two documents at a time.
Comment From: jego026
I added these 2 dependencies and the error went away:
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core</artifactId>
<version>1.55.4</version>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core-http-netty</artifactId>
<version>1.15.12</version>
</dependency>
Comment From: Javiervva
Hi, I'm getting the same error with OpenAI embeddings using the OpenSearch vector store on Windows.
This happens when calling vectorStore.add(docs);
Dependencies:
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-vector-store-opensearch</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-model-azure-openai</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-azure-store</artifactId>
<version>1.0.0</version>
</dependency>
Config:
spring:
ai:
vectorstore:
opensearch:
uris: http://localhost:9030
username: <username>
password: <password>
index-name: <index_name>
azure:
openai:
embedding:
options:
model: text-embedding-3-large
dimensions: 3072
user: <user>
Error:
java.lang.IllegalStateException: channel not registered to an event loop
at io.netty.channel.AbstractChannel.eventLoop(AbstractChannel.java:163) ~[netty-transport-4.1.122.Final.jar:4.1.122.Final]
at com.azure.core.http.netty.implementation.NettyUtility.closeConnection(NettyUtility.java:79) ~[azure-core-http-netty-1.15.11.jar:1.15.11]
at com.azure.core.http.netty.implementation.NettyAsyncHttpResponse.close(NettyAsyncHttpResponse.java:116) ~[azure-core-http-netty-1.15.11.jar:1.15.11]
at com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:249) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.http.policy.RetryPolicy.attemptSync(RetryPolicy.java:258) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.http.policy.RetryPolicy.processSync(RetryPolicy.java:161) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.http.policy.AddHeadersFromContextPolicy.processSync(AddHeadersFromContextPolicy.java:67) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.http.policy.RequestIdPolicy.processSync(RequestIdPolicy.java:77) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.implementation.http.rest.SyncRestProxy.send(SyncRestProxy.java:62) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.implementation.http.rest.SyncRestProxy.invoke(SyncRestProxy.java:83) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.implementation.http.rest.RestProxyBase.invoke(RestProxyBase.java:124) ~[azure-core-1.55.3.jar:1.55.3]
at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:95) ~[azure-core-1.55.3.jar:1.55.3]
at jdk.proxy2/jdk.proxy2.$Proxy171.getEmbeddingsSync(Unknown Source) ~[na:na]
at com.azure.ai.openai.implementation.OpenAIClientImpl.getEmbeddingsWithResponse(OpenAIClientImpl.java:2361) ~[azure-ai-openai-1.0.0-beta.16.jar:1.0.0-beta.16]
at com.azure.ai.openai.OpenAIClient.getEmbeddingsWithResponse(OpenAIClient.java:168) ~[azure-ai-openai-1.0.0-beta.16.jar:1.0.0-beta.16]
at com.azure.ai.openai.OpenAIClient.getEmbeddings(OpenAIClient.java:620) ~[azure-ai-openai-1.0.0-beta.16.jar:1.0.0-beta.16]
at org.springframework.ai.azure.openai.AzureOpenAiEmbeddingModel.lambda$call$1(AzureOpenAiEmbeddingModel.java:143) ~[spring-ai-azure-openai-1.0.0.jar:1.0.0]
at io.micrometer.observation.Observation.observe(Observation.java:564) ~[micrometer-observation-1.15.1.jar:1.15.1]
at org.springframework.ai.azure.openai.AzureOpenAiEmbeddingModel.call(AzureOpenAiEmbeddingModel.java:142) ~[spring-ai-azure-openai-1.0.0.jar:1.0.0]
at org.springframework.ai.embedding.EmbeddingModel.embed(EmbeddingModel.java:91) ~[spring-ai-model-1.0.0.jar:1.0.0]
at org.springframework.ai.vectorstore.opensearch.OpenSearchVectorStore.doAdd(OpenSearchVectorStore.java:207) ~[spring-ai-opensearch-store-1.0.0.jar:1.0.0]
at org.springframework.ai.vectorstore.observation.AbstractObservationVectorStore.lambda$add$1(AbstractObservationVectorStore.java:85) ~[spring-ai-vector-store-1.0.0.jar:1.0.0]
at io.micrometer.observation.Observation.observe(Observation.java:498) ~[micrometer-observation-1.15.1.jar:1.15.1]
at org.springframework.ai.vectorstore.observation.AbstractObservationVectorStore.add(AbstractObservationVectorStore.java:85) ~[spring-ai-vector-store-1.0.0.jar:1.0.0]
at com.mercedes.cart.intelligence.embedding.processor.DefaultEmbeddingProcessor.flushBatch(DefaultEmbeddingProcessor.java:72) ~[classes/:na]
at com.mercedes.cart.intelligence.embedding.processor.DefaultEmbeddingProcessor.lambda$process$0(DefaultEmbeddingProcessor.java:52) ~[classes/:na]
at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722) ~[na:na]
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) ~[na:na]
at com.mercedes.cart.intelligence.embedding.processor.DefaultEmbeddingProcessor.process(DefaultEmbeddingProcessor.java:45) ~[classes/:na]
at com.mercedes.cart.intelligence.embedding.listener.ApplicationReadyListener.onApplicationEvent(ApplicationReadyListener.java:27) ~[classes/:na]
at com.mercedes.cart.intelligence.embedding.listener.ApplicationReadyListener.onApplicationEvent(ApplicationReadyListener.java:14) ~[classes/:na]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185) ~[spring-context-6.2.8.jar:6.2.8]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178) ~[spring-context-6.2.8.jar:6.2.8]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156) ~[spring-context-6.2.8.jar:6.2.8]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:454) ~[spring-context-6.2.8.jar:6.2.8]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:387) ~[spring-context-6.2.8.jar:6.2.8]
at org.springframework.boot.context.event.EventPublishingRunListener.ready(EventPublishingRunListener.java:109) ~[spring-boot-3.5.3.jar:3.5.3]
at org.springframework.boot.SpringApplicationRunListeners.lambda$ready$6(SpringApplicationRunListeners.java:80) ~[spring-boot-3.5.3.jar:3.5.3]
at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118) ~[spring-boot-3.5.3.jar:3.5.3]
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112) ~[spring-boot-3.5.3.jar:3.5.3]
at org.springframework.boot.SpringApplicationRunListeners.ready(SpringApplicationRunListeners.java:80) ~[spring-boot-3.5.3.jar:3.5.3]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:332) ~[spring-boot-3.5.3.jar:3.5.3]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1361) ~[spring-boot-3.5.3.jar:3.5.3]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1350) ~[spring-boot-3.5.3.jar:3.5.3]
at com.mercedes.IntelligenceApplication.main(IntelligenceApplication.java:13) ~[classes/:na]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:50) ~[spring-boot-devtools-3.5.3.jar:3.5.3]
In my case, the error occurs intermittently. Yesterday it appeared when processing a smaller number of embeddings, and today it happens after processing some more, but none of the batches were particularly large. Do you know if this could be related to hitting some quota or limit?
Comment From: gbalboa-git
I had the same problem, not only with embeddings, also with chat models. intermittently "channel not registered to an event loop" errors. Very frustrating not having any leads from logs...
After some days and lots of research, I set the netty log level to debug (logging.level.reactor.netty: DEBUG) I realize that the error was indeed about quota limits of the models. In the logs the error was "Too many Requests" response from the Azure AI endpoint. I was reaching the RPM quota some times, and Max Tokens per minute some others...
It might be the same in your case. Hope it helps.