-
[x] I have checked that this issue has not already been reported.
-
[x] I have confirmed this bug exists on the latest version of pandas.
-
[x] (optional) I have confirmed this bug exists on the master branch of pandas.
Code Sample, a copy-pastable example
Hey--I noticed while running siuba's SQL unit tests that queries using the modulo operator are failing.
The issue is due to a recent change for issue #34211 in pandas setting the no_parameters argument before executing via a sqlalchemy engine. This was done to allow queries like SELECT 1 % 2, but causes SqlAlchemy expressions handling % to not always work with read_sql.
Solution. Rather than executing in a special way (no_parameters), I think you want to wrap a string query in SqlAlchemy.sql.text (see here). This will allow both queries with % and the full range of SqlAlchemy expressions. WDYT?
from sqlalchemy import sql, create_engine
import pandas as pd
engine = create_engine('postgresql://postgres:@localhost:5433/postgres', echo=False)
#engine = create_engine('postgresql://USERNAME:PASSWORD@localhost:PORT/DBNAME', echo=False)
# doesn't work, original issue in pandas: 'dict' object does not support indexing
engine.execute("SELECT 1 % 2")
# works, ideal solution
engine.execute(sql.text("SELECT 1 % 2" ))
# queries below broken by no_parameters change ----
pd.read_sql(sql.text("SELECT 1 % 2"), engine)
pd.read_sql(sql.select([sql.literal(1) % sql.literal(2)]), engine)
Here's a gist of the error for the last two queries...
ProgrammingError: (psycopg2.errors.UndefinedFunction) operator does not exist: integer %% integer
LINE 1: SELECT 1 %% 2
^
HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
[SQL: SELECT 1 %% 2]
(Background on this error at: http://sqlalche.me/e/13/f405)
Full traceback in the details
---------------------------------------------------------------------------
UndefinedFunction Traceback (most recent call last)
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/sqlalchemy/engine/base.py in _execute_context(self, dialect, constructor, statement, parameters, *args)
1267 self.dialect.do_execute_no_params(
-> 1268 cursor, statement, context
1269 )
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/sqlalchemy/engine/default.py in do_execute_no_params(self, cursor, statement, context)
595 def do_execute_no_params(self, cursor, statement, context=None):
--> 596 cursor.execute(statement)
597
UndefinedFunction: operator does not exist: integer %% integer
LINE 1: SELECT 1 %% 2
^
HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
The above exception was the direct cause of the following exception:
ProgrammingError Traceback (most recent call last)
<ipython-input-6-5831ea4e198c> in <module>
----> 1 pd.read_sql(sql.text("SELECT 1 % 2"), engine)
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/pandas/io/sql.py in read_sql(sql, con, index_col, coerce_float, params, parse_dates, columns, chunksize)
513 coerce_float=coerce_float,
514 parse_dates=parse_dates,
--> 515 chunksize=chunksize,
516 )
517
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/pandas/io/sql.py in read_query(self, sql, index_col, coerce_float, parse_dates, params, chunksize)
1293 args = _convert_params(sql, params)
1294
-> 1295 result = self.execute(*args)
1296 columns = result.keys()
1297
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/pandas/io/sql.py in execute(self, *args, **kwargs)
1160 """Simple passthrough to SQLAlchemy connectable"""
1161 return self.connectable.execution_options(no_parameters=True).execute(
-> 1162 *args, **kwargs
1163 )
1164
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/sqlalchemy/engine/base.py in execute(self, statement, *multiparams, **params)
2236
2237 connection = self._contextual_connect(close_with_result=True)
-> 2238 return connection.execute(statement, *multiparams, **params)
2239
2240 def scalar(self, statement, *multiparams, **params):
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/sqlalchemy/engine/base.py in execute(self, object_, *multiparams, **params)
1012 )
1013 else:
-> 1014 return meth(self, multiparams, params)
1015
1016 def _execute_function(self, func, multiparams, params):
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/sqlalchemy/sql/elements.py in _execute_on_connection(self, connection, multiparams, params)
296 def _execute_on_connection(self, connection, multiparams, params):
297 if self.supports_execution:
--> 298 return connection._execute_clauseelement(self, multiparams, params)
299 else:
300 raise exc.ObjectNotExecutableError(self)
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/sqlalchemy/engine/base.py in _execute_clauseelement(self, elem, multiparams, params)
1131 distilled_params,
1132 compiled_sql,
-> 1133 distilled_params,
1134 )
1135 if self._has_events or self.engine._has_events:
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/sqlalchemy/engine/base.py in _execute_context(self, dialect, constructor, statement, parameters, *args)
1316 except BaseException as e:
1317 self._handle_dbapi_exception(
-> 1318 e, statement, parameters, cursor, context
1319 )
1320
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/sqlalchemy/engine/base.py in _handle_dbapi_exception(self, e, statement, parameters, cursor, context)
1510 elif should_wrap:
1511 util.raise_(
-> 1512 sqlalchemy_exception, with_traceback=exc_info[2], from_=e
1513 )
1514 else:
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/sqlalchemy/util/compat.py in raise_(***failed resolving arguments***)
176
177 try:
--> 178 raise exception
179 finally:
180 # credit to
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/sqlalchemy/engine/base.py in _execute_context(self, dialect, constructor, statement, parameters, *args)
1266 if not evt_handled:
1267 self.dialect.do_execute_no_params(
-> 1268 cursor, statement, context
1269 )
1270 else:
~/Dropbox/Repo/siuba/env/lib/python3.6/site-packages/sqlalchemy/engine/default.py in do_execute_no_params(self, cursor, statement, context)
594
595 def do_execute_no_params(self, cursor, statement, context=None):
--> 596 cursor.execute(statement)
597
598 def is_disconnect(self, e, connection, cursor):
ProgrammingError: (psycopg2.errors.UndefinedFunction) operator does not exist: integer %% integer
LINE 1: SELECT 1 %% 2
^
HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
[SQL: SELECT 1 %% 2]
(Background on this error at: http://sqlalche.me/e/13/f405)
Comment From: jbrockmendel
@machow want to make a PR for this?
Comment From: machow
@jbrockmendel sure! I have the changes running and passing tests in the pandas docker image. I'm noticing that the tests cover only either queries with parameters or using % as an operator.
I'll try adding two new kinds of tests:
- queries with
%%(should pass after changes) - queries with a combination of parameters and
%as operator. (should fail before and after changes?).
Will add tests and open a PR tomorrow.
Comment From: jreback
might be closed by https://github.com/pandas-dev/pandas/pull/37534 (e.g. working on master).
Comment From: machow
Hey--that PR (#37534) just changed the code in PR #34211 back, so reverted back to pandas v1.05 behavior that I listed in my PR (#36275).
Here is the table I made in that PR for reference
| query | v1.05 | v.1.1 | my PR #36275 | issue |
|---|---|---|---|---|
no params, % |
❌ | ✅ | ✅ | #34211 (merged) |
no params, %% |
✅ | ❌ | ❌ | #35871 |
params, % |
❌ | ❌ | ❌ | https://github.com/psycopg/psycopg2/issues/827 |
params, %% |
✅ | ✅ | ✅ | |
sqla declarative, % |
✅ | ❌ | ✅ | #35484 |
That means this code worked in v1.1, but fails now (as it did in v1.05; it's the no params, % case):
import pandas as pd
from sqlalchemy import create_engine
# 1.3.5
pd.__version__
engine = create_engine("postgresql://postgres:@localhost:5432/postgres", echo=False)
# TypeError: dict is not a sequence
pd.read_sql("SELECT 1 % 2", engine)
I'm not sure whether y'all consider this a bug or not, but pandas' behavior as it exists now IMO aligns well with sqlalchemy's API, since technically you shouldn't pass a sql string to engine.execute, but wrap it in sqlalchemy.sql.text (which is how you can support people passing SQL as a string).
Comment From: adejumoridwan
@machow I would like to take this issue, I am a new contributor. Probably need some guidance.
Comment From: machow
Hey! I haven't looked at this issue for a while, and am not sure if it's intended behavior in pandas or not. I wonder if another issue with the label "good first issue" might be a bit easier to pick up?
Comment From: VedangPokharkar
take
Comment From: VedangPokharkar
hello, @machow , is this issue still being pursued?
Comment From: sandervh14
I just encountered this too. This bug prevents executing SELECT ... WHERE col like '...%...' statements.
Comment From: emilenstrom
I just encountered this too. This bug prevents executing
SELECT ... WHERE col like '...%...'statements.
The fix for this is to use two %, example:
SELECT ... WHERE fullname like 'John%%'
instead of SELECT ... WHERE fullname like 'John%'
Read about it here: https://stackoverflow.com/questions/64252764/sql-case-when-x-like-t-in-python-script-resulting-in-typeerror-dict-is-not
Comment From: adarsh3690704
engine.execute(r"SELECT 1 % 2")
how about writing like this, converting it to raw string , if i understand the problem right
Comment From: riteshpen
I want to work on this issue has it already been resolved.
Comment From: samyarpotlapalli
take