So I’m no expert, but I have been a hobbyist C and Rust dev for a while now, and I’ve installed tons of programs from GitHub and whatnot that required manual compilation or other hoops to jump through, but I am constantly befuddled installing python apps. They seem to always need a very specific (often outdated) version of python, require a bunch of venv nonsense, googling gives tons of outdated info that no longer works, and generally seem incredibly not portable. As someone who doesn’t work in python, it seems more obtuse than any other language’s ecosystem. Why is it like this?
No, it’s not just you, Python’s tooling is a mess. It’s not necessarily anyone’s fault but there are a ton of options and a lot of very similarly naked things that accomplish different (but sometimes similar) tasks. As someone who considers themselves between beginner and intermediate proficiency in Python this is my biggest hurdle right now.
Python’s tooling is a mess.
Not only that. It’s a historic mess. Over the years, growing a better and better toolset left a lot of projects in a very messy state. So many answers on Stack Overflow that mention
easy_install- I still don’t know what it is, but I guess it was some kind of protouv.Every time I’m doing anything with Python I ask myself if Java’s tooling is this complicated or I’m just used to it by now. I think a big part of the weirdness is that a lot of Python tooling is tied to the Python installation whereas in Java things like Maven and Gradle are separate. In addition, I think dependencies you install get tied to that Python installation, while in Java they just are in a cache for Maven/Gradle. And in the horrible scenario where you need to use different versions of Maven/Gradle (one place I was at specifically needed Maven 3.0.3 for one project and a different for a different, don’t ask, it’s dumb and their own fault for setting it up that way) at least they still have one common cache for everything.
I guess it also helps that with Java you (often) don’t need platform specific jar files. But Python is often used as an easy and dynamic scripting interface over more performant, native code. So you don’t really run into things like “this artifact doesn’t have a 64 bit arm version for python 2” often with Java. But that’s not a fault of Python’s tooling, it’s just the reality of how it’s used.
Python is the only programming language that has forced me to question what the difference is between an egg and a wheel.
Python developer here. Venv is good, venv is life. Every single project I create starts with
python3 -m venv venv
source venv/bin/activate
pip3 install {everything I need}
pip3 freeze > requirements.txt
Now write code!
Don’t forget to update your requirements.txt using pip3 freeze again anytime you add a new library with pip.
If you installed a lot of packages before starting to develop with virtual environments, some libraries will be in your OS python install and won’t be reflected in pip freeze and won’t get into your venv. This is the root of all evil. First of all, don’t do that. Second, you can force libraries to install into your venv despite them also being in your system by installing like so:
pip3 install --ignore-installed mypackage
If you don’t change between Linux and windows most libraries will just work between systems, but if you have problems on another system, just recreate the whole venv structure
rm -rf venv (…make a new venv, activate it) pip3 install -r requirements.txt
Once you get the hang of this you can make Python behave without a lot of hassle.
This is a case where a strength can also be a weakness.
You have been in lala land for too long. That list of things to do is insane. Venv is possibly one of the worst solutions around, but many Python devs are incapable of seeing how bad it is. Just for comparison, so you can understand, in Ruby literally everything you did is covered by one command
bundle. On every system.OP sounds like a victim of Python 3, finding various Python 2 projects on the internet, a venv isn’t going to help
pip3 freeze > requirements.txt
I hate this. Because now I have a list of your dependencies, but also the dependencies of the dependencies, and I now have regular dependencies and dev-dependencies mixed up. If I’m new to Python I would have NO idea which libraries would be the important ones because it’s a jumbled mess.
I’ve come to love
uv(coming frompoetry, coming frompipwith arequirements/base.txtandrequirements/dev.txt- gotta keep regular dependencies and dev-dependencies separate).uv syncuv run <application>That’s it. I don’t even need to install a compatible Python version, as
uvtakes care of that for me. It’ll automatically create a local.venv/, and it’s blazingly fast.I’ve never really spent much time with uv, I’ll give it a try. It seems like it takes a few steps out of the process and some guesswork too.
Okay, now give me those steps but what to do if I clone an already existing repo please
The git repo should ignore the venv folder, so when you clone you then create a new one and activate it with those steps.
Then when you are installing requirements with pip, the repo you cloned will likely have a requirements.txt file in it, so you ‘pip install -r requirements.txt’
This is the way
This is the way.
Just out of curiosity, I haven’t seen anyone recommend miniconda… Why so, is there something wrong I’m not aware of?
I’m no expert, but I totally feel you, python packages, dependencies and version matching is a real nightmare. Even with
venvI had a hard time to make everything work flawlessly, especially on MacOS.However, with miniconda everything was way easier to configure and worked as expected.
Isn’t conda specifically for mathy things?
Yes it’s terrible. The only hope on the horizon is
uv. It’s significantly better than all the other tooling (Poetry, pip, pipenv, etc.) so I think it has a good chance of reducing the options to just Pip oruvat least.But I fully expect the Python Devs to ignore it, and maybe even make life deliberately difficult for it like they did for static analysers. They have some strange priorities sometimes.
uv is good but it needs a little more time in the oven.
For the moment I would definitely recommend poetry if you are not a library developer. Poetry’s biggest sin is it’s atrocious performance but it has most of the features you need to work with Python apps today.
Why do you say it needs more time in the oven? I’ve had zero issues with it as a drop-in replacement for Pip in a large commercial project, which is an extremely impressive achievement. (And it was 10x faster.)
I tried Poetry once and it failed to resolve dependencies on the first thing I tried it on. If anything Poetry needs more time in the oven. It also wasn’t 10x faster.
UV is a game changer for python.
I hated the tooling until I found it.
I like the idea of
uv, but I hate the name. Libuv is already a very popular C library, and used in everything from NodeJS to Julia to Python (through the popularuvloopmodule). Every time I see someone mentionuvI get confused and think they’re talking about uvloop until I remember the Astral project, and then reconfirm to myself how much I disapprove of their name choice.I don’t think
libuvis really that popular, nor is it that confusing.But I do agree it’s not a very good name. “Rye” is a much better name. Probably too late anyway.
The venv stuff is pretty annoying, I agree.
As a baby Python Dev, I’m glad it’s not just me.
I’ve been full time writing python professionally since 2015. You get used to it. It starts to just make sense and feel normal. Then when you move to a different language environment you wonder why their tooling doesn’t use a virtualenv.
No it’s not. E.g. nobody who starts a new project uses setup.py anymore
Are you sure? I’m not very active in that ecosystem, but if that was prevalent in the past, surely there’s still tutorials and stuff out there that people would follow and create such projects even today?
More than that, it seems to me that the official python docs for packaging [still] talks about setup.py. Why would people not use that?
Sure, there was some hyperbole. Some people need some specific setuptools plugin or something. Almost nobody.
when the official docs are telling you to use it, then it’s used. You can have no expectation of people to think the tooling isn’t shit when it’s literally the official recommendation.
OP seems to be trying to install older projects, rather than creating a new project.
Yep, they are not portable, every app should come bundled with its own interpreter. As to why, I think historically it didn’t target production grade application development.
I’m not sure this can be really fixed with Python 3, maybe we just have to hope for Python 4
It’s fixed, and the python version had nothing to do with it. Just use hatch
Ah yes, the 15th standard we’ve been waiting for!
It’s not a standard, it’s built on standards.
You can also use Poetry (which recently grew standard metadata support) or plain
uv venvif you want to do things manually but fast.Just use this one… or any of this 4 others.
This is the issue for us, python outsiders. Each time we try we get a different answer with new tools. We are outside of the comtunity, we don’t know the trend, old and new, pro and cons.
Your first recommandation is hatch… first time I’ve heard of it. Uv seems trendy in this thread, but before that it was unknown to me too.
As I understands it, it should be pip’s job. When it detect I’m in a project it install packages in it and python use them. It can use any tool under the hood, but the default package manager shoud be able to do it on its own.
Uv and pip do the same thing, uv is just faster.
Hatch has the same role as Poetry or tox: managing environments for you.
Applications should be packaged properly, in a self contained installer for exactly this demographic. It’s not Python’s fault that this isn’t common practice.
I’ve started using poetry and the experience has improved.
The reason you do stuff in a venv is to isolate that environment from other python projects on your system, so one Python project doesn’t break another. I use Docker for similar reasons for a lot of non-Python projects.
A lot of Python projects involve specific versions of libraries, because things break. I’ve had similar issues with non-Python projects. I’m not sure I’d say Python is particularly worse about it.
There are tools in place that can make the sharing of Python projects incredibly easy and portable and consistent, but I only ever see the best maintained projects using them unfortunately.
Python is hacky, because it hacks. There’s a bunch of ways you can do anything. You can run it on numerous platforms, or even on web assembly. It’s not maintained centrally. Each “app” you find is just somebodies hack project they’re sharing with you for fun.
Python is the new Perl
On that note, I’m hesitant between writing my scripts in perl or python right now. Bash prevent sharing with Windows peoples… I just want to provide easy wrappers tools that are usually aroud 10 lines of shell, but testers ain’t on linux so they cannot use them.
I don’t know perl, but each time I interract with pyton’s projects I have a different venv/poetry/… to setup. Forget adout it the next time and nothing is kept easy to reuse.
Perl isn’t really any better. There aren’t easy tools that do the same thing as venv. They exist, but they are not easy. Plus there are a much larger amount of cpan modules that have c in them than python.
After using python, I’m of the opinion that perl was much cleaner.
Nothing comes close to Perl’s abuse of global variables. Oh you called this function? Take a guess which global variables it will use.
Docker might be solution here.
But from my experience most python scripts are absolute junk. The barrier for entry is low so there’s a massive disparity in quality.
Python is truly a mess when Docker is considered a solution.
Difficult? How so? I find compiling C and C++ stuff much more difficult than anything python. It never works on the first try whereas with python the chances are much much higher.
What’s is so difficult to understand about virtual envs? You have global python packages, you can also have per user python packages, and you can create virtual environments to install packages into. Why do people struggle to understand this?
The global packages are found thanks to default locations, which can be overridden with environment variables. Virtual environments set those environment variables to be able to point to different locations.
python -m venv .venv/means python will execute the modulevenvand tell it to create a virtual environment in the.venvfolder in the current directory. As mentioned above, the environment variables have to be set to actually use it. That’s whensource .venv/bin/activatecomes into play (there are other scripts for zsh and fish). Now you can runpip install $packageand then run the package’s command if it has one.It’s that simple. If you want to, you can make it difficult by doing
sudo pip install $packageand fucking up your global packages by possibly updating a dependency of another package - just like the equivalent of updating glibc from 1.2 to 1.3 and breaking every application depending on 1.2 because glibc doesn’t fucking follow goddamn semver.As for old versions of python, bro give me a break. There’s pyenv for that if whatever old ass package you’re installing depends on an ancient 10 year old python version. You really think building a C++ package from 10 years ago will work more smoothly than python? Have fun tracking down all the unlocked dependency versions that “Worked On My Machine 🏧” at the start of the century.
The only python packages I have installing are those with C/C++ dependencies which have to be compiled at install time.
Y’all have got to be meme’ing.
I think you have got to be meme’ing. You literally wrote 7 paragraphs about how to build something for python when for other languages it’s literally a single command. For Ruby, it’s literally
bundle. Nothing else. Doesn’t matter if it’s got C packages or not. Doesn’t matter if it’s windows or not. Doesn’t matter if you have a different project one folder over that uses an older gem or not. Doesn’t matter if it’s 15 years old or not. One command.Just for comparison for gradle it’s
./gradlew buildFor maven ismvn installFor Elixir it’smix deps.getmix compileFor node it’snpm installevery other language it’s hardly more than 1 command.
Python is the only language that thinks that it’s even slightly acceptable to have virtual environments when it was universally decided upon decades ago to be a tremendously bad idea. Just like node_modules which also was known to be a bad idea before npm decided to try it out again, only for it to be proven to be a bad idea right off the bat. And all the other python build tools have agreed that virtual envs are bad.
everyone focuses on the tooling, not many are focusing on the reason: python is extremely dynamic. like, magic dynamic you can modify a module halfway through an import, you can replace class attributes and automatically propagate to instances, you can decompile the bytecode while it’s running.
combine this with the fact that it’s installed by default and used basically everywhere and you get an environment that needs to be carefully managed for the sake of the system.
js has this packaging system down pat, but it has the advantage that it got mainstream in a sandboxed isolated environment before it started leaking out into the system. python was in there from the beginning, and every change breaks someone’s workflow.
the closest language to look at for packaging is probably lua, which has similar issues. however since lua is usually not a standalone application platform it’s not a big deal there.
and yet that all works fine in Ruby, which came out around the same time as Python and yet has had Bundler for 15 years now.
Python - 15+ package managers and build tools Ruby - 1
the closest language to look at for packaging is probably lua, which has similar issues. however since lua is usually not a standalone application platform it’s not a big deal there.
no the closest language is literally Ruby, it’s almost the exact same language, except the tooling isn’t insane and it came out only a few years after python.
good point, ruby is a good comparison. although, ruby is very different under the hood. it’s magically dynamic in a completely different way, and it also never really got the penetration on the system level that python did.
none of this is to take away from the fact that python packaging is bad. i know how to work it because i’ve been programming in python for 14 years, but trying to teach people makes the problem obvious. and yet.
