Opened 2 years ago

Closed 22 months ago

#10950 closed bug (fixed)

Sphinx "RecursionError: maximum recursion depth exceeded while pickling an object"

Reported by: ezyang Owned by:
Priority: normal Milestone: 8.0.1
Component: Documentation Version: 7.11
Keywords: Cc: olsner, slyfox
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: Compile-time crash Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s): Phab:D1809
Wiki Page:

Description

During a recent validate with Sphinx (sphinx-build) 1.3.1 (as packaged by Arch Linux), I got this failure:

reading sources... [ 33%] glasgow_exts                                         
Exception occurred:
  File "/usr/lib/python3.5/site-packages/sphinx/environment.py", line 863, in read_doc
    pickle.dump(doctree, f, pickle.HIGHEST_PROTOCOL)
RecursionError: maximum recursion depth exceeded while pickling an object
The full traceback has been saved in /tmp/sphinx-err-11x82xjc.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
docs/users_guide/ghc.mk:30: recipe for target 'docs/users_guide/ghc.1' failed
make[1]: *** [docs/users_guide/ghc.1] Error 1
Makefile:121: recipe for target 'all' failed
make: *** [all] Error 2

The contents of the referenced log:

# Sphinx version: 1.3.1  
# Python version: 3.5.0 (CPython)  
# Docutils version: 0.12 release  
# Jinja2 version: 2.8  
# Last messages:  
#   reading sources... [  6%] bugs  
#   reading sources... [  9%] codegens  
#   reading sources... [ 12%] debugging  
#   reading sources... [ 15%] editing-guide  
#   reading sources... [ 18%] extending_ghc  
#   reading sources... [ 21%] ffi-chap  
#   reading sources... [ 24%] flags  
#   reading sources... [ 27%] ghc  
#   reading sources... [ 30%] ghci  
#   reading sources... [ 33%] glasgow_exts  
# Loaded extensions:
#   sphinx.ext.extlinks (1.3.1) from /usr/lib/python3.5/site-packages/sphinx/ext/extlinks.py
#   alabaster (0.7.6) from /usr/lib/python3.5/site-packages/alabaster/__init__.py
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/sphinx/cmdline.py", line 245, in main
    app.build(opts.force_all, filenames)
  File "/usr/lib/python3.5/site-packages/sphinx/application.py", line 264, in build
    self.builder.build_update()
  File "/usr/lib/python3.5/site-packages/sphinx/builders/__init__.py", line 240, in build_update
    self.build(['__all__'], to_build)  
  File "/usr/lib/python3.5/site-packages/sphinx/builders/__init__.py", line 259, in build
    self.doctreedir, self.app))  
  File "/usr/lib/python3.5/site-packages/sphinx/environment.py", line 618, in update
    self._read_serial(docnames, app)
  File "/usr/lib/python3.5/site-packages/sphinx/environment.py", line 638, in _read_serial
    self.read_doc(docname, app)
  File "/usr/lib/python3.5/site-packages/sphinx/environment.py", line 863, in read_doc
    pickle.dump(doctree, f, pickle.HIGHEST_PROTOCOL)
RecursionError: maximum recursion depth exceeded while pickling an object

Change History (15)

comment:1 Changed 2 years ago by Ben Gamari <ben@…>

In fa5eabe/ghc:

sphinx: Don't share doctrees between targets

Sphinx may trip over itself when multiple instances are run in parallel.

Fixes #10950.

comment:2 Changed 2 years ago by thomie

Milestone: 8.0.1
Resolution: fixed
Status: newclosed

Assuming this is fixed.

comment:3 Changed 2 years ago by bgamari

I was waiting to hear back from ezyang before closing but it's quite likely that this is fixed.

comment:4 Changed 2 years ago by ezyang

Sorry! I haven't seen the error recently, so looks OK.

comment:5 Changed 2 years ago by olsner

Cc: olsner added

I started getting this error when running validate today (I last ran validate sometime before the doc changes). This is with sphinx 1.2.2 from Ubuntu's python3-sphinx package.

If you retry once or twice with ./validate --no-clean, that usually gets past it though.

comment:6 Changed 2 years ago by ezyang

Resolution: fixed
Status: closednew

Yeah, I've seen it too. Reopening.

comment:7 Changed 2 years ago by olsner

Adding the -E option to sphinx ("don't use a saved environment, always read all files") seems to work around it for me, presumably because it disables the code that is trying to pickle stuff.

EDIT: Nope, wasn't enough. Even with the -E I got this error from validate again.

Last edited 2 years ago by olsner (previous) (diff)

comment:8 Changed 2 years ago by slyfox

Cc: slyfox added

comment:9 Changed 2 years ago by Sergei Trofimovich <siarheit@…>

In a034031/ghc:

extending_ghc.rst: fix broken link (Trac #10950)

The error exibits as build failures
of two types:

  1. extending_ghc.rst:: ERROR:
      Anonymous hyperlink mismatch:
        1 references but 0 targets.
      See "backrefs" attribute for IDs.

  2. reading sources... [ 33%] glasgow_exts
       Exception occurred:
          pickle.dump(doctree, f, pickle.HIGHEST_PROTOCOL)
       RecursionError: maximum recursion depth exceeded
       while pickling an object

Broken link created circular reference and failed to
serialize the result.

Fixed the problem by pointing to relevant section.

Signed-off-by: Sergei Trofimovich <siarheit@google.com>

comment:10 Changed 2 years ago by slyfox

Resolution: fixed
Status: newclosed

I think empty link in form of

`foo <>`__

was able to make sphinx try to serialize bad parse state.

Should be good now.

comment:11 Changed 22 months ago by slyfox

Resolution: fixed
Status: closednew

Issue still persists in -HEAD. I think it manifests better with unfixed warnings.

Another stab at it:

https://phabricator.haskell.org/D1809

comment:12 Changed 22 months ago by slyfox

Differential Rev(s): Phab:D1809

comment:13 Changed 22 months ago by Sergei Trofimovich <siarheit@…>

In 4c11db6/ghc:

sphinx-build: fix python stack overflow (Trac #10950)

Summary:
commit a034031a102bc08c76a6cdb104b72922ae22c96b did not
fix problem completely. Stack overflows still occasionally
happen.

Easy to test by the following Torture Test:

  while sphinx-build -T -N -E -a -b html \
        -d docs/users_guide/.doctrees-html \
        -D latex_paper_size=letter \
        docs/users_guide docs/users_guide/build-html/users_guide
  do
    echo again
  done

sphinx build large nested data structures when parses GHC manual
(docs/users_guide/glasgow_exts.rst is 455KB in size) which
can't be serialized useing default python call stack depth of 1000
calls.

The patch increases stack depth 10 times. Survived 2 hours of
Torture Test.

Signed-off-by: Sergei Trofimovich <siarheit@google.com>

Test Plan: ran Torture Test to make sure it is stable

Reviewers: austin, bgamari

Reviewed By: bgamari

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1809

GHC Trac Issues: #10950

comment:14 Changed 22 months ago by slyfox

Status: newmerge

comment:15 Changed 22 months ago by bgamari

Resolution: fixed
Status: mergeclosed
Note: See TracTickets for help on using tickets.