Readline issues on new install of FreeBSD 13.1

Too many issues will kill our team's development velocity, drastically.
Make sure you have checked all steps below.

Prerequisite

Make sure your problem is not listed in the common build problems.
Make sure no duplicated issue has already been reported in the pyenv issues. You should look for closed issues, too.
Make sure you are not asking us to help solving your specific issue.
- GitHub issues is opened mainly for development purposes. If you want to ask someone to help solving your problem, go to some community site like Gitter, StackOverflow, etc.
Make sure your problem is not derived from packaging (e.g. Homebrew).
- Please refer to the package documentation for the installation issues, etc.
Make sure your problem is not derived from plugins.
- This repository is maintaining pyenv and the default python-build plugin only. Please refrain from reporting issues of other plugins here.

Description

Platform information (e.g. Ubuntu Linux 16.04): FreeBSD 13.1-RELEASE-p3
OS architecture (e.g. amd64): amd64
pyenv version: pyenv 2.3.9
Python version: Python 3.9.16 (installed via pkg)
C Compiler information (e.g. gcc 7.3): FreeBSD clang version 13.0.0
Please attach the debug trace of the failing command as a gist:
- Run env PYENV_DEBUG=1 <faulty command> 2>&1 | tee trace.log and attach trace.log. E.g. if you have a problem with installing Python, run env PYENV_DEBUG=1 pyenv install -v <version> 2>&1 | tee trace.log (note the -v option to pyenv install).

Recently provisioned a remote VPS on FreeBSD 13.1 and wanted to give pyenv a whirl. Seems I'm hit with readline issues. I attempted to install readline libs, but what I believe to be happening is that pyenv make cannot resolve readline. Here is what I performed:

installed pyenv via bash script
Ran:

pyenv install 3.11.1

Errors reported:

pyenv: /home/blogger/.pyenv/versions/3.11.1 already exists
continue with installation? (y/N) y
Downloading Python-3.11.1.tar.xz...
-> https://www.python.org/ftp/python/3.11.1/Python-3.11.1.tar.xz
Installing Python-3.11.1...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'readline'
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?
Installed Python-3.11.1 to /home/blogger/.pyenv/versions/3.11.1

Packages installed:

# pkg info | grep -i read
readline-8.2.0                 Library for editing command lines as they are typed
tclreadline-2.3.8              GNU readline for TCL

What I believe is happening is that GNU readline requirement is not being read properly as the output shows:

checking how to link readline libs... none

Debug log attached.

trace.log

Could you please also attach config.log from the build directory?

Do you have Readline development headers and lib files installed? Through what means? (I dunno if BSD have package managers. I know they have "ports" which are kinda that and may need additional compiler flags to link against.)

The config.log file does not appear to be created:

[~/.pyenv] $ find . | grep config.log (returns no results)

I made sure to install readline libs using pkg command as stated in my first update. Just to be sure, I logged on and verified:

# pkg info readline
readline-8.2.0
Name           : readline
Version        : 8.2.0
Installed on   : Fri Jan  6 12:31:28 2023 EST
Origin         : devel/readline
Architecture   : FreeBSD:13:amd64
Prefix         : /usr/local
Categories     : devel
Licenses       : GPLv3
Maintainer     : sunpoet@FreeBSD.org
WWW            : https://tiswww.case.edu/php/chet/readline/rltop.html
Comment        : Library for editing command lines as they are typed
Options        :
        BRACKETEDPASTE : on
        DOCS           : on
Shared Libs provided:
        libreadline.so.8
        libhistory.so.8
Annotations    :
        FreeBSD_version: 1301000
        cpe            : cpe:2.3:a:gnu:readline:8.2:::::freebsd13:x64
        repo_type      : binary
        repository     : FreeBSD
Flat size      : 1.78MiB
Description    :
The GNU Readline library provides a set of functions for use by applications
that allow users to edit command lines as they are typed in. Both Emacs and vi
editing modes are available. The Readline library includes additional functions
to maintain a list of previously-entered command lines, to recall and perhaps
reedit those lines, and perform csh-like history expansion on previous commands.

WWW: https://tiswww.case.edu/php/chet/readline/rltop.html

# find /usr/local/ | grep readline
/usr/local/share/licenses/readline-8.2.0
/usr/local/share/licenses/readline-8.2.0/catalog.mk
/usr/local/share/licenses/readline-8.2.0/LICENSE
/usr/local/share/licenses/readline-8.2.0/GPLv3
/usr/local/share/licenses/tclreadline-2.3.8
/usr/local/share/licenses/tclreadline-2.3.8/catalog.mk
/usr/local/share/licenses/tclreadline-2.3.8/LICENSE
/usr/local/share/licenses/tclreadline-2.3.8/BSD3CLAUSE
/usr/local/share/doc/readline
/usr/local/share/doc/readline/CHANGES
/usr/local/share/doc/readline/INSTALL
/usr/local/share/doc/readline/README
/usr/local/share/readline
/usr/local/share/readline/excallback.c
/usr/local/share/readline/fileman.c
/usr/local/share/readline/histexamp.c
/usr/local/share/readline/manexamp.c
/usr/local/share/readline/rl-fgets.c
/usr/local/share/readline/rl.c
/usr/local/share/readline/rlbasic.c
/usr/local/share/readline/rlcat.c
/usr/local/share/readline/rlevent.c
/usr/local/share/readline/rlptytest.c
/usr/local/share/readline/rltest.c
/usr/local/share/readline/rlversion.c
/usr/local/share/info/readline.info
/usr/local/share/vim/vim90/indent/readline.vim
/usr/local/share/vim/vim90/ftplugin/readline.vim
/usr/local/share/vim/vim90/syntax/readline.vim
/usr/local/include/readline
/usr/local/include/readline/chardefs.h
/usr/local/include/readline/history.h
/usr/local/include/readline/keymaps.h
/usr/local/include/readline/readline.h
/usr/local/include/readline/rlconf.h
/usr/local/include/readline/rlstdc.h
/usr/local/include/readline/rltypedefs.h
/usr/local/include/readline/tilde.h
/usr/local/include/editline/readline.h
/usr/local/include/tclreadline.h
/usr/local/lib/libreadline.a
/usr/local/lib/libreadline.so
/usr/local/lib/libreadline.so.8
/usr/local/lib/libreadline.so.8.2
/usr/local/lib/python3.9/lib-dynload/readline.cpython-39.so
/usr/local/lib/python3.9/lib2to3/fixes/__pycache__/fix_xreadlines.cpython-39.opt-1.pyc
/usr/local/lib/python3.9/lib2to3/fixes/__pycache__/fix_xreadlines.cpython-39.opt-2.pyc
/usr/local/lib/python3.9/lib2to3/fixes/__pycache__/fix_xreadlines.cpython-39.pyc
/usr/local/lib/python3.9/lib2to3/fixes/fix_xreadlines.py
/usr/local/lib/python3.9/test/__pycache__/test_readline.cpython-39.opt-1.pyc
/usr/local/lib/python3.9/test/__pycache__/test_readline.cpython-39.opt-2.pyc
/usr/local/lib/python3.9/test/__pycache__/test_readline.cpython-39.pyc
/usr/local/lib/python3.9/test/test_readline.py
/usr/local/lib/libtclreadline.so
/usr/local/lib/libtclreadline-2.3.8.so
/usr/local/lib/tclreadline2.3.8
/usr/local/lib/tclreadline2.3.8/pkgIndex.tcl
/usr/local/lib/tclreadline2.3.8/tclreadlineCompleter.tcl
/usr/local/lib/tclreadline2.3.8/tclreadlineInit.tcl
/usr/local/lib/tclreadline2.3.8/tclreadlineSetup.tcl
/usr/local/libdata/pkgconfig/readline.pc
/usr/local/man/man3/readline.3.gz
/usr/local/man/man3/libssh2_knownhost_readline.3.gz
/usr/local/man/mann/tclreadline.n.gz

config.log is in the build directory under /tmp. In the log, it's /tmp/python-build.20230106142056.35755.

Log attached.
python-build.20230107112237.54254.log

I said config.log, not the build log.
It should be under the build directory, e.g. /tmp/python-build.20230106142056.35755/config.log.

I removed all files from /tmp and this is the only log that is being generated.

Oh, that's right. Your build succeeds, so the build path is being removed.

Please run pyenv install with -k to keep it.

It appears config.log was generated in another location with the -k switch:

[~] $ find . | grep config.log
./.pyenv/sources/3.11.1/Python-3.11.1/config.log

Please see attached. I am seeing linker errors, so am assuming would need to append something to either LDFLAGS or CFLAGS. Let me know.

config.log

Okay, it looks like unlike Linux, in BSD, the compiler is configured to not look in /usr/local/ by default:

configure:20925: checking how to link readline libs
configure:20952: cc -pthread -o conftest  -I/home/blogger/.pyenv/versions/3.11.1/include  -L/home/blogger/.pyenv/versions/3.11.1/lib -Wl,-rpath=/home/blogger/.pyenv/versions/3.11.1/lib  conftest.c -lreadline -ldl  -lutil >&5
ld: error: unable to find library -lreadline
cc: error: linker command failed with exit code 1 (use -v to see invocation)
configure:20952: $? = 1
configure: failed program was:
<...>
| char readline ();
| int
| main ()
| {
| return readline ();
|   ;
|   return 0;
| }

Could you test the codebase from https://github.com/native-api/pyenv/tree/use_freebsd_ports ?

Using your code tree the latest python available is 3.9.7. Is there anyway I can test 3.11.1?

Use the use_freebsd_ports branch

Same error. Looks like the same output:

I rebuilt to regenerate relevant logs. Please see attached.

[~/.pyenv] $ git branch
  master
* use_freebsd_ports

$ pyenv install 3.11.1 -k
Installing Python-3.11.1...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'readline'
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?
Installed Python-3.11.1 to /home/blogger/.pyenv/versions/3.11.1

config.log

Okay, I was able to set up a FreeBSD VM and reproduce the issue.

It turns out, you don't have pkg-config installed.

configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

Which, counterintuitively, is in the pkgconf Ports package.

All ports modules have .pc files that allow to locate them using pkg-config (which Python's Configure uses as well).

We can still add compiler flags for the case when readline package is installed and pkg-config is missing.
I've checked that either way works.

Great find. I installed pkgconf, the replacement for pkg-config but still getting a failure. I'm not using any ports and do not even have the ports tree populated, only binary pacckages. I'm still receiving the error. Please see config.log.

config.log

Ok this seems to have gotten me past readline:

[H: blog] [U: blogger] [~/.pyenv] $ git branch
  master
* use_freebsd_ports

[H: blog] [U: blogger] [~/.pyenv] $ git log --pretty=format:'%h' -n 1
08835d22

[H: blog] [U: blogger] [~/.pyenv] $ find /usr/local/libdata/pkgconfig/ | grep readline
/usr/local/libdata/pkgconfig/readline.pc

[H: blog] [U: blogger] [~/.pyenv] $ pyenv install 3.11.1 -k
Installing Python-3.11.1...
/home/blogger/.pyenv/plugins/python-build/bin/python-build: line 1425: dev/null: No such file or directory
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  File "/home/blogger/.pyenv/versions/3.11.1/lib/python3.11/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?
Installed Python-3.11.1 to /home/blogger/.pyenv/versions/3.11.1

[H: blog] [U: blogger] [~/.pyenv] $ pyenv shell 3.11.1
[H: blog] [U: blogger] [~/.pyenv] $ python3.11
Python 3.11.1 (main, Jan  8 2023, 15:13:27) [Clang 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a3 on freebsd13
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
>>>

Let me get sqlite up before we close this. Would love a complete build. Thanks!

I'm wondering if there needs to be some pkgconf optimizations to look for libraries in /usr/local/lib:

[H: blog] [U: blogger] [~/.pyenv] $ pkgconf --libs readline
-L/usr/local/lib -lreadline
[H: blog] [U: blogger] [~/.pyenv] $ pkgconf --libs sqlite
-L/usr/local/lib -lsqlite
[H: blog] [U: blogger] [~/.pyenv] $ pkg-config --libs readline
-L/usr/local/lib -lreadline
[H: blog] [U: blogger] [~/.pyenv] $ pkg-config --libs sqlite
-L/usr/local/lib -lsqlite

Is there a way to get the build script to use pkgconf/pkg-config by default on FreeBSD?

CPython's Configure is supposed to already be using pkg-config to locate libraries.

Ok troubleshooting this further, with the pkgconf/pkg-config commands it seems the build script expects sqlite3. The version I had installed was sqlite, which defaults to 2 on FreeBSD.

$ pyenv install 3.11.1 -k
Installing Python-3.11.1...
/home/blogger/.pyenv/plugins/python-build/bin/python-build: line 1425: dev/null: No such file or directory
Installed Python-3.11.1 to /home/blogger/.pyenv/versions/3.11.1

$ pkgconf --libs readline
-L/usr/local/lib -lreadline

$ pkgconf --libs sqlite3
-L/usr/local/lib -lsqlite3

$ pyenv shell 3.11.1

$ python3.11
Python 3.11.1 (main, Jan  8 2023, 15:28:35) [Clang 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a3 on freebsd13
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'sqlite'
>>> import sqlite3
>>> import readline

I think we are good here. Believe we can close this. Definitely appreciate your help. One last question, will these changes be merged into master pyenv?

I grabbed latest merge today, 2369ac2a089e1afaf61753ee1d4bc08b80c1ace7 and I receive the same readline error, although sqlite is fine. I cannot compare the tree with master since it has been deleted. Can we double check?
config.log

Testing with 2.3.10

Same readline error. Please reopen.

Okay, it turns out that -- surprisingly enough -- Python's Configure does NOT use pkg-config to locate Readline.
And the PR's code only worked because my invocation for command -v pkg-config was broken! So when I fixed it when merging...

I have been testing on a new VM and it does appear that the crux of this issue is python not using pkg-config for readLine.

Interestingly enough, specifying libedit does allow it to build, as well as setting flags manually. I believe the only way around this would be to just set the flags in the previous fix if FreeBSD is detected. I did read up on this and it does appear that you are correct in that tool chains don't look at /usr/local.

What really bothers me about this is that there IS a readline.pc file available in PKG_CONFIG_PATH. Think only this could be fixed upstream.

I've tested that https://github.com/native-api/pyenv/tree/fix_freebsd fixes the probem for me (the URL means https://github.com/native-api/pyenv repo, fix_freebsd branch). Could you confirm?

What really bothers me about this is that there IS a readline.pc file available in PKG_CONFIG_PATH. Think only this could be fixed upstream.

Yes, this is an upstream problem.
They already fixed it in 3.12.0a1.

Ok this branch works. I have also confirmed pkg info -e lib returns 0, so it should pass that test.

Navigation Menu

Repository navigation

Readline issues on new install of FreeBSD 13.1 #2585

Prerequisite

Description

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Readline issues on new install of FreeBSD 13.1 #2585

Description

Prerequisite

Description

Activity

native-api commented on Jan 6, 2023

dcontiveros commented on Jan 6, 2023

native-api commented on Jan 7, 2023

dcontiveros commented on Jan 7, 2023

native-api commented on Jan 7, 2023

dcontiveros commented on Jan 7, 2023

native-api commented on Jan 7, 2023

dcontiveros commented on Jan 7, 2023

native-api commented on Jan 7, 2023

native-api commented on Jan 7, 2023

dcontiveros commented on Jan 7, 2023

native-api commented on Jan 7, 2023

dcontiveros commented on Jan 7, 2023

native-api commented on Jan 7, 2023

dcontiveros commented on Jan 8, 2023

1 remaining item

dcontiveros commented on Jan 8, 2023

dcontiveros commented on Jan 8, 2023

native-api commented on Jan 8, 2023

dcontiveros commented on Jan 8, 2023

dcontiveros commented on Jan 10, 2023

dcontiveros commented on Jan 10, 2023

dcontiveros commented on Jan 10, 2023

native-api commented on Jan 13, 2023

dcontiveros commented on Jan 13, 2023

native-api commented on Jan 13, 2023

native-api commented on Jan 13, 2023

dcontiveros commented on Jan 13, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions