Sphinx "RecursionError: maximum recursion depth exceeded while pickling an object"

[ezyang]

During a recent validate with Sphinx (sphinx-build) 1.3.1 (as packaged by Arch Linux), I got this failure:

reading sources... [ 33%] glasgow_exts                                         
Exception occurred:
  File "/usr/lib/python3.5/site-packages/sphinx/environment.py", line 863, in read_doc
    pickle.dump(doctree, f, pickle.HIGHEST_PROTOCOL)
RecursionError: maximum recursion depth exceeded while pickling an object
The full traceback has been saved in /tmp/sphinx-err-11x82xjc.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
docs/users_guide/ghc.mk:30: recipe for target 'docs/users_guide/ghc.1' failed
make[1]: *** [docs/users_guide/ghc.1] Error 1
Makefile:121: recipe for target 'all' failed
make: *** [all] Error 2

The contents of the referenced log:

# Sphinx version: 1.3.1  
# Python version: 3.5.0 (CPython)  
# Docutils version: 0.12 release  
# Jinja2 version: 2.8  
# Last messages:  
#   reading sources... [  6%] bugs  
#   reading sources... [  9%] codegens  
#   reading sources... [ 12%] debugging  
#   reading sources... [ 15%] editing-guide  
#   reading sources... [ 18%] extending_ghc  
#   reading sources... [ 21%] ffi-chap  
#   reading sources... [ 24%] flags  
#   reading sources... [ 27%] ghc  
#   reading sources... [ 30%] ghci  
#   reading sources... [ 33%] glasgow_exts  
# Loaded extensions:
#   sphinx.ext.extlinks (1.3.1) from /usr/lib/python3.5/site-packages/sphinx/ext/extlinks.py
#   alabaster (0.7.6) from /usr/lib/python3.5/site-packages/alabaster/__init__.py
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/sphinx/cmdline.py", line 245, in main
    app.build(opts.force_all, filenames)
  File "/usr/lib/python3.5/site-packages/sphinx/application.py", line 264, in build
    self.builder.build_update()
  File "/usr/lib/python3.5/site-packages/sphinx/builders/__init__.py", line 240, in build_update
    self.build(['__all__'], to_build)  
  File "/usr/lib/python3.5/site-packages/sphinx/builders/__init__.py", line 259, in build
    self.doctreedir, self.app))  
  File "/usr/lib/python3.5/site-packages/sphinx/environment.py", line 618, in update
    self._read_serial(docnames, app)
  File "/usr/lib/python3.5/site-packages/sphinx/environment.py", line 638, in _read_serial
    self.read_doc(docname, app)
  File "/usr/lib/python3.5/site-packages/sphinx/environment.py", line 863, in read_doc
    pickle.dump(doctree, f, pickle.HIGHEST_PROTOCOL)
RecursionError: maximum recursion depth exceeded while pickling an object

[Ben Gamari <ben@…>]

In fa5eabe/ghc:

sphinx: Don't share doctrees between targets

Sphinx may trip over itself when multiple instances are run in parallel.

Fixes #10950.

[thomie]

Assuming this is fixed.

[bgamari]

I was waiting to hear back from ezyang before closing but it's quite likely that this is fixed.

[ezyang]

Sorry! I haven't seen the error recently, so looks OK.

[olsner]

I started getting this error when running validate today (I last ran validate sometime before the doc changes). This is with sphinx 1.2.2 from Ubuntu's python3-sphinx package.

If you retry once or twice with ./validate --no-clean, that usually gets past it though.

[ezyang]

Yeah, I've seen it too. Reopening.

[olsner]

Adding the -E option to sphinx ("don't use a saved environment, always read all files") seems to work around it for me, presumably because it disables the code that is trying to pickle stuff.

EDIT: Nope, wasn't enough. Even with the -E I got this error from validate again.

[Sergei Trofimovich <siarheit@…>]

In a034031/ghc:

extending_ghc.rst: fix broken link (Trac #10950)

The error exibits as build failures
of two types:

  1. extending_ghc.rst:: ERROR:
      Anonymous hyperlink mismatch:
        1 references but 0 targets.
      See "backrefs" attribute for IDs.

  2. reading sources... [ 33%] glasgow_exts
       Exception occurred:
          pickle.dump(doctree, f, pickle.HIGHEST_PROTOCOL)
       RecursionError: maximum recursion depth exceeded
       while pickling an object

Broken link created circular reference and failed to
serialize the result.

Fixed the problem by pointing to relevant section.

Signed-off-by: Sergei Trofimovich <siarheit@google.com>

[slyfox]

I think empty link in form of

`foo <>`__

was able to make sphinx try to serialize bad parse state.

Should be good now.

[slyfox]

Issue still persists in -HEAD. I think it manifests better with unfixed warnings.

Another stab at it:

​https://phabricator.haskell.org/D1809

[Sergei Trofimovich <siarheit@…>]

In 4c11db6/ghc:

sphinx-build: fix python stack overflow (Trac #10950)

Summary:
commit a034031a102bc08c76a6cdb104b72922ae22c96b did not
fix problem completely. Stack overflows still occasionally
happen.

Easy to test by the following Torture Test:

  while sphinx-build -T -N -E -a -b html \
        -d docs/users_guide/.doctrees-html \
        -D latex_paper_size=letter \
        docs/users_guide docs/users_guide/build-html/users_guide
  do
    echo again
  done

sphinx build large nested data structures when parses GHC manual
(docs/users_guide/glasgow_exts.rst is 455KB in size) which
can't be serialized useing default python call stack depth of 1000
calls.

The patch increases stack depth 10 times. Survived 2 hours of
Torture Test.

Signed-off-by: Sergei Trofimovich <siarheit@google.com>

Test Plan: ran Torture Test to make sure it is stable

Reviewers: austin, bgamari

Reviewed By: bgamari

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1809

GHC Trac Issues: #10950

[bgamari]

Merged to ghc-8.0 as e6bbaefcf7d0aa9ce5d6845cf82329e473fd88c0.