I want to use the gemini apiKey to access my gemini Ai instead of using project-id+location.
Currently, the spring ai module using gemini only supports project-id+location using google cloud. The apiKey mode in Google ai studio is not supported.
My request was to support the apiKey pattern, similar to openai.
Comment From: tzolov
Hey @yangfeng20 could you provide us with a the Gemini documentation that explains how to create and use api keys?
Comment From: yangfeng20
Ok, please refer to: https://ai.google.dev/gemini-api/docs/api-overview https://ai.google.dev/gemini-api/docs/api-key https://ai.google.dev/gemini-api/docs/get-started/rest
Comment From: vanduc2514
@yangfeng20 There is a way to work around with the current spring-ai-vertex-ai-gemini dependency by replacing the low-level Request Formatter in HttpJsonPredictionServiceStub with the one pointing to the api endpoint of new Gemini AI
ProtoMessageRequestFormatter.<GenerateContentRequest>newBuilder()
.setPath("/v1beta/models/{model=*}:generateContent", request -> Map.of("model", "gemini-pro"))
.setQueryParamsExtractor(request -> Map.of("key", List.of("<GEMINI_API_KEY>")))
.setRequestBodyExtractor(request -> ProtoRestSerializer.create().toBody("*", request.toBuilder().clearModel().build(),false))
.build()
then set the VertexAI endpoint to https://generativelanguage.googleapis.com and use Transport.REST for VertexAI transport. The above code snippet only works for generateContent endpoint. For other endpoints to work, you will need to replace other request formatters. It's ok for the time being since Google has not provided an offical Java depdency yet.
Comment From: marcosamm
@yangfeng20 There is a way to work around with the current
spring-ai-vertex-ai-geminidependency by replacing the low-level Request Formatter inHttpJsonPredictionServiceStubwith the one pointing to the api endpoint of new Gemini AI
java ProtoMessageRequestFormatter.<GenerateContentRequest>newBuilder() .setPath("/v1beta/models/{model=*}:generateContent", request -> Map.of("model", "gemini-pro")) .setQueryParamsExtractor(request -> Map.of("key", List.of("<GEMINI_API_KEY>"))) .setRequestBodyExtractor(request -> ProtoRestSerializer.create().toBody("*", request.toBuilder().clearModel().build(),false)) .build()then set the VertexAI endpoint to
https://generativelanguage.googleapis.comand useTransport.RESTfor VertexAI transport. The above code snippet only works forgenerateContentendpoint. For other endpoints to work, you will need to replace other request formatters. It's ok for the time being since Google has not provided an offical Java depdency yet.
@vanduc2514 , I didn't understand. Where do I need to set the transport to REST? Could you provide a short functional example using ProtoMessageRequestFormatter and setting Transport.REST?
Comment From: vanduc2514
@marcosamm Sorry for making not clear. The new version of Spring AI 1.0.0-SNAPSHOT does not require the Transport.REST to be set. To implement this work-around you will need to create a custom implementation of PredictionServiceStub usingProtoMessageRequestFormatterthat points to the Gemini endpoint. Then use it for the VertexAiGeminiChatModel.
Here is an example for how to do it
https://gist.github.com/vanduc2514/0a74ec0ec160538c39313e73f27f8740
Then you will use the VertexAIAdapter class from the gist instead of the default VertexAI
@Bean
ChatModel chatModel() {
var vertexAIAdapter = new VertexAIAdapter("gemini-base-url", "gemini-api-key")
return new VertexAiGeminiChatModel(vertexAIAdapter);
}
You also need to disable VertexAiGeminiAutoConfiguration for skipping the auto bean creation.
Comment From: marcosamm
@vanduc2514 , this works for me.
Sometimes I get timeout exceptions like this:
java.net.SocketTimeoutException: Read timed out
at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:278) ~[na:na]
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304) ~[na:na]
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346) ~[na:na]
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796) ~[na:na]
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099) ~[na:na]
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489) ~[na:na]
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483) ~[na:na]
at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1461) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066) ~[na:na]
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:291) ~[na:na]
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347) ~[na:na]
at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420) ~[na:na]
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399) ~[na:na]
at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:827) ~[na:na]
at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:759) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1690) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[na:na]
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531) ~[na:na]
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307) ~[na:na]
at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36) ~[google-http-client-1.44.2.jar:1.44.2]
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:152) ~[google-http-client-1.44.2.jar:1.44.2]
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[google-http-client-1.44.2.jar:1.44.2]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.44.2.jar:1.44.2]
at com.google.api.gax.httpjson.HttpRequestRunnable.run(HttpRequestRunnable.java:115) ~[gax-httpjson-2.49.0.jar:2.49.0]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
But it's normal in more complex interactions.
Thank you very much!
Comment From: vanduc2514
@marcosamm You are welcome, I'm glad that it works for you. The issue you have seems related to SSL handshake timeout, does it frequently happen ?
Comment From: markpollack
Hi. I spoke with the google team, this support is supposed to be coming in a newer version of their Java SDK. I'd rather wait for them to provide this than introduce workarounds. I'm frustrated as well as it prevents us from creating integration tests on github actions.
Comment From: yangfeng20
Thanks for your reply, which means that the gemini api scheduling will be implemented in spring ai using google's sdk. Thank you for your previous efforts. Can you release the code you wrote earlier as a third-party project? I want to learn from it. Thank you.
Comment From: marcosamm
@marcosamm You are welcome, I'm glad that it works for you. The issue you have seems related to SSL handshake timeout, does it frequently happen ?
Sorry for the delay. Yes, it happens frequently, especially when many tokens are sent.
Comment From: account123456789
@vanduc2514 , this works for me.
Sometimes I get timeout exceptions like this:
java.net.SocketTimeoutException: Read timed out at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:278) ~[na:na] at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304) ~[na:na] at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346) ~[na:na] at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796) ~[na:na] at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099) ~[na:na] at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489) ~[na:na] at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483) ~[na:na] at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70) ~[na:na] at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1461) ~[na:na] at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066) ~[na:na] at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:291) ~[na:na] at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347) ~[na:na] at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420) ~[na:na] at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399) ~[na:na] at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:827) ~[na:na] at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:759) ~[na:na] at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1690) ~[na:na] at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[na:na] at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531) ~[na:na] at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307) ~[na:na] at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36) ~[google-http-client-1.44.2.jar:1.44.2] at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:152) ~[google-http-client-1.44.2.jar:1.44.2] at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[google-http-client-1.44.2.jar:1.44.2] at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.44.2.jar:1.44.2] at com.google.api.gax.httpjson.HttpRequestRunnable.run(HttpRequestRunnable.java:115) ~[gax-httpjson-2.49.0.jar:2.49.0] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[na:na] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]But it's normal in more complex interactions.
Thank you very much!
Any Solution for the timeout issue , bro ?? I am facing it too much with Gemini unfortunately.
Comment From: dhananjay-personatech
Thanks @vanduc2514 , for VertexAIAdapter. With this adapter, conversation history is not getting stored and used. Can you please suggest change to support AI Advisors?
Comment From: markpollack
There has been a long history around using just an API key to access gemini, historically a split in offerings with different APIs. There is now a new SDK from googel that removes this split. We will address this in a post GA release.
Comment From: yangfeng20
Alright, thank you for the response and support from your team. May I ask when this version is expected to be released approximately?
Comment From: Alos
On a related topic, how do I disable Vertex while running tests in my Github Action?
Factory method 'connectionDetails' threw exception with message: Vertex AI project-id must be set!
I added the following to the boostrap.properties:
spring.ai.model.chat=none
spring.ai.vertex.ai.embedding.text=none
spring.ai.vertex.ai.embedding.multimodal=none