google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute' #104
Comments
|
I also tried to look at the GCP > IAM & Admin > Quotas limits for Spanner but the admin one can't be modified. |
|
Here is a more complete traceback: Looking at it it seems that during database = instance.database(database_id, pool=pool)
if not database.exists(): # <<<- admin call to Spanner
raise ValueError("database '%s' does not exist." % database_id)
conn = Connection(instance, database)
if pool is not None:
conn._own_pool = False
return connand this is probably what triggers the admin quota limit. I tried to reproduce the same error using with database.snapshot() as snapshot:
results = snapshot.execute_sql()But I don't see the error. This makes me think that somehow, |
|
When commenting: database = instance.database(database_id, pool=pool)
# if not database.exists(): # <<<- admin call to Spanner
# raise ValueError("database '%s' does not exist." % database_id)I don't see the error anymore. |
|
In case it's useful for someone else, here is a monkey patch that works for me: def _patch_spanner_connect():
"""Prevent admin quot limits.
See https://github.com/cloudspannerecosystem/python-spanner-sqlalchemy/issues/104
"""
from google.cloud import spanner_dbapi
from google.cloud.spanner_dbapi import Connection
from google.api_core.gapic_v1.client_info import ClientInfo
from google.cloud import spanner_v1 as spanner
from google.cloud.spanner_dbapi.version import DEFAULT_USER_AGENT
from google.cloud.spanner_dbapi.version import PY_VERSION
def _patched_connect(
instance_id,
database_id,
project=None,
credentials=None,
pool=None,
user_agent=None,
):
client_info = ClientInfo(user_agent=user_agent or DEFAULT_USER_AGENT, python_version=PY_VERSION)
if isinstance(credentials, str):
client = spanner.Client.from_service_account_json(
credentials, project=project, client_info=client_info
)
else:
client = spanner.Client(project=project, credentials=credentials, client_info=client_info)
instance = client.instance(instance_id)
if not instance.exists():
raise ValueError("instance '%s' does not exist." % instance_id)
database = instance.database(database_id, pool=pool)
# NOTE(hadim): monkey patch to remove Spanner admin calls.
# if not database.exists():
# raise ValueError("database '%s' does not exist." % database_id)
conn = Connection(instance, database)
if pool is not None:
conn._own_pool = False
return conn
spanner_dbapi.connect = _patched_connect |
|
@hadim, thank you very much for your investigation! I think you're right, |
|
I've taken a look at the Database class, and I don't see what can be used instead of
Seems to me the best way to resolve the problem is to move instance and database existence check into a separate As an alternative, we probably can use a simple select from the given database in order to check that it's is reachable. In any case I think some kind of existance check should be there. @olavloite, what do you think? |
|
Java (JDBC) does not really do a check like this, so if the user connects to a non-existing database, the error will be delayed until the first request that is done. I guess the closest we have in any of the other connection APIs is the That method executes a |
|
@olavloite, I suppose in Spanner terms Then, manual method with |
product-auto-label
bot
added
the
api: spanner
I am running into some quota limitations when doing a high number of read requests in parallel:
The request is quite similar to a simple
SELECT * FROM table1 as t1 JOIN table2 as t2 ON t1.id = t2.id.The error says Administrative requests per minute which seems weird to me since it has nothing to do with an admin action right?
Is it something related to
python-spanner-sqlalchemy?From https://cloud.google.com/spanner/quotas#administrative_limits, it seems like the limit can't be modified.
This issue is quite limiting for us at the moment to scale our workload with Spanner.
The text was updated successfully, but these errors were encountered: