[1.11.x] Fixed #28062 -- Added a setting to disable server-side cursors on PostgreSQL.

When a connection pooler is set up in transaction pooling mode, queries
relying on server-side cursors fail. The DISABLE_SERVER_SIDE_CURSORS
setting in DATABASES disables server-side cursors for this use case.

Backport of 88336fdbb5 from master

django/db/models/query.py

@@ -319,7 +319,8 @@ class QuerySet(object):
         An iterator over the results from applying this QuerySet to the
         database.
         """
-        return iter(self._iterable_class(self, chunked_fetch=True))
+        use_chunked_fetch = not connections[self.db].settings_dict.get('DISABLE_SERVER_SIDE_CURSORS')
+        return iter(self._iterable_class(self, chunked_fetch=use_chunked_fetch))
 
     def aggregate(self, *args, **kwargs):
         """

docs/ref/databases.txt

@@ -189,6 +189,41 @@ cursor query is controlled with the `cursor_tuple_fraction`_ option.
 
 .. _cursor_tuple_fraction: https://www.postgresql.org/docs/current/static/runtime-config-query.html#GUC-CURSOR-TUPLE-FRACTION
 
+.. _transaction-pooling-server-side-cursors:
+
+Transaction pooling and server-side cursors
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 1.11.1
+
+Using a connection pooler in transaction pooling mode (e.g. `pgBouncer`_)
+requires disabling server-side cursors for that connection.
+
+Server-side cursors are local to a connection and remain open at the end of a
+transaction when :setting:`AUTOCOMMIT <DATABASE-AUTOCOMMIT>` is ``True``. A
+subsequent transaction may attempt to fetch more results from a server-side
+cursor. In transaction pooling mode, there's no guarantee that subsequent
+transactions will use the same connection. If a different connection is used,
+an error is raised when the transaction references the server-side cursor,
+because server-side cursors are only accessible in the connection in which they
+were created.
+
+One solution is to disable server-side cursors for a connection in
+:setting:`DATABASES` by setting :setting:`DISABLE_SERVER_SIDE_CURSORS
+<DATABASE-DISABLE_SERVER_SIDE_CURSORS>` to ``True``.
+
+To benefit from server-side cursors in transaction pooling mode, you could set
+up :doc:`another connection to the database </topics/db/multi-db>` in order to
+perform queries that use server-side cursors. This connection needs to either
+be directly to the database or to a connection pooler in session pooling mode.
+
+Another option is to wrap each ``QuerySet`` using server-side cursors in an
+:func:`~django.db.transaction.atomic` block, because it disables ``autocommit``
+for the duration of the transaction. This way, the server-side cursor will only
+live for the duration of the transaction.
+
+.. _pgBouncer: https://pgbouncer.github.io/
+
 Test database templates
 -----------------------
 

docs/ref/models/querysets.txt

@@ -2041,6 +2041,11 @@ won't cache results after iterating over them. Oracle and :ref:`PostgreSQL
 <postgresql-server-side-cursors>` use server-side cursors to stream results
 from the database without loading the entire result set into memory.
 
+On PostgreSQL, server-side cursors will only be used when the
+:setting:`DISABLE_SERVER_SIDE_CURSORS <DATABASE-DISABLE_SERVER_SIDE_CURSORS>`
+setting is ``False``. Read :ref:`transaction-pooling-server-side-cursors` if
+you're using a connection pooler configured in transaction pooling mode.
+
 .. versionchanged:: 1.11
 
     PostgreSQL support for server-side cursors was added.

docs/ref/settings.txt

@@ -646,6 +646,21 @@ PostgreSQL), it is an error to set this option.
 
 When :setting:`USE_TZ` is ``False``, it is an error to set this option.
 
+.. setting:: DATABASE-DISABLE_SERVER_SIDE_CURSORS
+
+``DISABLE_SERVER_SIDE_CURSORS``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 1.11.1
+
+Default: ``False``
+
+Set this to ``True`` if you want to disable the use of server-side cursors with
+:meth:`.QuerySet.iterator`. :ref:`transaction-pooling-server-side-cursors`
+describes the use case.
+
+This is a PostgreSQL-specific setting.
+
 .. setting:: USER
 
 ``USER``

docs/releases/1.11.1.txt

@@ -4,7 +4,17 @@ Django 1.11.1 release notes
 
 *Under development*
 
-Django 1.11.1 fixes several bugs in 1.11.
+Django 1.11.1 adds a minor feature and fixes several bugs in 1.11.
+
+Allowed disabling server-side cursors on PostgreSQL
+===================================================
+
+The change in Django 1.11 to make :meth:`.QuerySet.iterator()` use server-side
+cursors on PostgreSQL prevents running Django with `pgBouncer` in transaction
+pooling mode. To reallow that, use the :setting:`DISABLE_SERVER_SIDE_CURSORS
+<DATABASE-DISABLE_SERVER_SIDE_CURSORS>` setting in :setting:`DATABASES`.
+
+See :ref:`transaction-pooling-server-side-cursors` for more discussion.
 
 Bugfixes
 ========

docs/spelling_wordlist

@@ -581,6 +581,8 @@ plugins
 pluralizations
 po
 podcast
+pooler
+pooling
 popup
 postfix
 postgis

tests/backends/test_postgresql.py

@@ -1,5 +1,7 @@
+import operator
 import unittest
 from collections import namedtuple
+from contextlib import contextmanager
 
 from django.db import connection
 from django.test import TestCase
@@ -23,6 +25,18 @@ class ServerSideCursorsPostgres(TestCase):
             cursors = cursor.fetchall()
         return [self.PostgresCursor._make(c) for c in cursors]
 
+    @contextmanager
+    def override_db_setting(self, **kwargs):
+        for setting, value in kwargs.items():
+            original_value = connection.settings_dict.get(setting)
+            if setting in connection.settings_dict:
+                self.addCleanup(operator.setitem, connection.settings_dict, setting, original_value)
+            else:
+                self.addCleanup(operator.delitem, connection.settings_dict, setting)
+
+            connection.settings_dict[setting] = kwargs[setting]
+            yield
+
     def test_server_side_cursor(self):
         persons = Person.objects.iterator()
         next(persons)  # Open a server-side cursor
@@ -52,3 +66,17 @@ class ServerSideCursorsPostgres(TestCase):
         del persons
         cursors = self.inspect_cursors()
         self.assertEqual(len(cursors), 0)
+
+    def test_server_side_cursors_setting(self):
+        with self.override_db_setting(DISABLE_SERVER_SIDE_CURSORS=False):
+            persons = Person.objects.iterator()
+            next(persons)  # Open a server-side cursor
+            cursors = self.inspect_cursors()
+            self.assertEqual(len(cursors), 1)
+            del persons  # Close server-side cursor
+
+        with self.override_db_setting(DISABLE_SERVER_SIDE_CURSORS=True):
+            persons = Person.objects.iterator()
+            next(persons)  # Should not open a server-side cursor
+            cursors = self.inspect_cursors()
+            self.assertEqual(len(cursors), 0)