Repository navigation
Python 3 todo list #5
Description
I've broken up python 3 migration (#1) into the following independent tasks.
Fix the (python 2) skipped tests see #3:
- mysql (this just needs travis to install it I think this is no big deal, i.e. just a line to .travis.yml)
- test command line (uses PIPE atm which doesn't work on py2.6)
- test _keywords
- skip MediaWikiArticle
- I think there is a couple more from travis passing python 2 tests #3 (search for FIXME and TODO)
Python 3 stuff
- print statements (Future stage1 python3 #4)
- get imports working from Future stage1 python3 #4
- sgmllib is depreciated in python 3 (!) https://docs.python.org/2/library/sgmllib.html (I've kinda worked around this by renaming html.parser (which is a hack), the fact html.parser is in base may mean this section of pattern is no longer required sgmllib is deprecated clips/pattern#4
- ... (triage remaining)
test files passing on python 3
e.g. via nosetests test/test_xx.py
- add separate travis py3 build for the thus far passing tests
- test_graph TST graph tests pass #11
- test_text en de... text passing py3 tests #16
- test_metric
- test_vector Vector #17
- test_web
- test_db
- remove py3 special case build (once all the above are passing!)
General stuff
- use a single README (probably an rst is best)
- add a MANIFEST.in - burn with fire the current os.walk stuff (other files?)
- grab version number from the init file (rather than the pattern installation)
- style is a bit random for importing py3 stuff atm (my fault) should be more consistent
- remove testing
suitefunctions (just usetest_main). - some tests are pretty flaky (ie. numbers changing)
- remove utf-8 print hack added to the examples (due to
__future__ print_statementbeing sensitive to unicode) i.e. make all the things unicode. See Unicode all the things #12 - some tests are ombscurified by being so class based (although some which are still not very dry), atm I prefixed these with Abstract... but they should probably be ABC (the key is they can't start with Test otherwise nose etc tries to run them - and fails).
- javascript tests (?) currently not run - unclear if do they do anything?!
- add examples to tests (e.g. have just run all the example py files, no assertions just running... but potentially could add some asserts?) - I nearly have done this.
- work out which dependancies can stop being vendorized (see below)
- depreciation messages of uses of dependancies (which have been updated), see travis/nose output
- add travis, coveralls, landscape (sign up with pattern & pattern3 gh accounts and add this repo)
- add banner for the above into README
- pep8/docformatter all the things (this should be simultaneously merged with clips/pattern, otherwise merging in work to clips will be very very difficult ???)
- decide on when this can be merged back upstream (IMO this should be asap, we don't require py3 to be ready just that py2 still works... and a quick pep8 storm :) )
Performace
- come up with some benchmarks to compare python 2 and 3 (and potentially the old code base)
- profile and see what can be improved...
Activity
hayd commentedon Oct 29, 2014
I would also like to do a big (edit: automated) pep8 cleanup, but really that requires downing-tools on the main pattern repo (until this effort is ready to merge back in), as merge-conflicts after a pep8 storm can be expensive (so you want to do it quickly). Need confirmation that this is the game plan.
hayd commentedon Oct 29, 2014
One big problem IMO is vendorising all these modules, which are clearly dependancies e.g. beautiful soup, simplejson, feedparser?!!? Waaat. These should all be removed and added as dependencies.
We don't want to port those or manage them... at all.
(needs python3 fixes, but they have tests + travis!) Author claims they will never release a python 3 version, but there is a fork/PR Python 3.4 compatibility, and nosetests euske/pdfminer#71 ... it may beeasier to use pdfminer if py2, pdfminer3k if py3 (at least for now) :shayd commentedon Oct 30, 2014
@tom-de-smedt I have a branch with this running all 485 tests in python3 (and passing in python 2 as well, obviously the python 3 tests fail). Would be great if you could look at / merge the pending PRs.
I ripped out all the dependancies (as discussed above), it was less painful than I thought it would be (as well as being "the right thing to do"). As such the upcoming PR is somewhat larger (and potentially more controversial)...
waylonflinn commentedon Oct 31, 2014
Thanks @hayd! You've done an impressive amount of stuff in the past couple of days! I'll take a look at those pull requests tomorrow morning and get them merged in.
hayd commentedon Nov 1, 2014
Fab, thanks!
I just want to clarify my thoughts for moving forward, since it's still unclear how large a job it'll be to fully migrate we want to minimize git-pain later on...
merge all the PRs (above) ;)pep8 clean up (happy to do this, it's just a script)This way any bug fixing merged/PRd on clips/pattern during python 3 development can be merged cleanly (after the 1000s of lines py3 syntax and pep8 changes).
Update: I feel less strongly about this now.
EnricoGiampieri commentedon Apr 9, 2015
Hi, I'm interested in helping the port. I routinely work in python3 and I'm really interested in a "modern" version of this library. Any suggestion on where to start from?
hayd commentedon Apr 9, 2015
@EnricoGiampieri I think getting
test_db.pyto run successfully is next up. Run the test and fix the errors/failures :)wannaphong commentedon Dec 11, 2015
fix sgmllib is depreciated in python 3 #21
maxschaf commentedon Jul 27, 2016
hi, there was no action in this repo during the last year. anyone still working on this?
souravsingh commentedon Sep 10, 2016
Hi, I would like to contribute to the Project by fixing some skipped tests. How do I start?
hayd commentedon Sep 11, 2016
@souravsingh see the comments in #3. larger projects are test_web and test_db.
#17 was waiting on @tom-de-smedt, so this port has stalled.
james-see commentedon Sep 19, 2016
@hayd for mysql, I would just add in mysqlclient as a requirement, so in travis.yml add in install: pip3 install mysqlclient (or add it to requirements.txt) mysqlclient is a drop in replacement for MySQLdb and even imports the same way as import MySQLdb.
james-see commentedon Sep 19, 2016
@hayd how best for me to do this part myself? fork and pull request it for that specific issue?
JimSEOW commentedon Nov 22, 2016
As part of the process of running Gensim in py3.5 Win64 Windows10, I made a few changes to pattern3
piskvorky/gensim#1025
Update [1st Dec 2016]
@i git clone https://github.com/pattern3/pattern; python setup.py develop in MiniConda py3.5 Win64
=> First problem: pypiwin32
=> I downloaded pypiwin32-219-cp35-none-win_amd64.whl
=> pip install pypiwin32-219-cp35-none-win_amd64.whl
=> repeat python setup.py develop with pattern3
=> Install works ok
FYI: I have not done all test yet!
utkarshrai commentedon Mar 6, 2017
Is the project still on? I'd love to join in?
city-in-the-sky commentedon May 31, 2017
Update: As part of Google Summer of Code 2017, Markus Beuckelmann (@markus-beuckelmann) will be working on the future of Pattern (porting it to Python 3 is first on our list). Markus is admin of the main repo now and can handle pull requests and invite collaborators. Be sure to reach out to him and include him in discussions about the port. We'll focus on the main repo, and reintegrate what we can from the work done by @hayd here. Hopefully we can make some progress over the summer.