unlike the pharma companies.

Apparently pharma companies don’t like their drugs being used for executions. This has lead to a number of states having to try to buy black market supplies to comply with existing written state laws for drugs used and other states change their drug cocktail to alternatives with a number of disastrous effects.

Kind of like “compressed”. It takes longer/more effort to run them, to produce the same result as a the same model that has not been quantized, where that non-quantized version would consume significantly more RAM but produce the result faster. You would typically only run the quantized model when you’re starved for RAM, which most of us are running LLMs locally.

Think like zipping a file with file compression. It takes less space, but has to be unzipped for you to have usable files again.

“Weights” in LLMs are the “final answer” numbers from the results of model training, and these are the engine of the LLM model. Lets use chocolate chip cookies as an analogy.

The chocolate chip cookies are produced with ingredients, a recipe, labor effort to produce the dough, and then cooking energy/effort to bake chocolate chip cookies. In a traditional AI company, the company gets the ingredients, they write their own recipe, do all the dough creation, and then the energy/labor for baking, and charge you money to get chocolate chip cookies.

An open-weights AI company got all the ingredients, wrote their own recipe, labor effort to produce the dough, and instead of charging you for the dough, you get as much uncooked dough as you want forever. Your only task is to take the dough and cook it yourself and you’ve got free chocolate chip cookies. You can make as many cookies as you want with your own oven. However, you are not given free raw ingredients, nor are your given the recipe to alter it in a way you might like. You can only get the dough for free.

So open-weight AI models (uncooked dough) are LLMs you can use on your own computers (oven) for free and have LLM output ( chocolate chip cookies), but you don’t get the training data (ingredients) nor the training parameters (recipe) that built the open-weight model.

Question, what do you do with it

I use it for both fun throwaway stuff as well as productivity tasks for learning.

Example of fun throwaway:

Do you remember that episode of Seinfeld where the Kramer starts using Facebook marketplace and starts buying the most worthless items before being robbed when trying to get a too-good-to-be-true sale? No? Because it never happened, but you can plug that premise into an crafted LLM prompt and it will pop out a whole TV script with in-character dialog for each actor as well as use of popular existing sets.

Foreign language learning:

I’m studying a foreign language and want to interact with just the level of vocabulary and grammar I have knowledge of right now at my level for practice. I can prompt the LLM to limit itself to just what I know now and adjust the conversation level so I can practice. If I ever get stuck, I can ask the LLM to explain the grammar usage or vocabulary choice.

and how is the data that underlies the model harvested?

I’m not sure what you’re asking here. Are you asking, for example, how the Deepseek model was trained? If so, I answered that above. If not, can you rephrase your question?

The absolutely most generous copyright protection length is in Mexico which is 100 years plus the life of the author when created. So lets generously say a total of 160 years. Most of the rest of the world is 120 year max. Anything outside of that is in the Public Domain. Many things had a much shorter path to the Public Domain falling into it in as little as 20 years.

So lots and lots of stuff is not copyrighted.

As an exercise so I could learn the technology, I trained a model exclusively on the Public Domain works of L Frank Baum, specifically the “Wonderful Wizard of Oz” series (did you know there are 14 books in the series just by Baum?!). The model created is not useful as a tool, and more often than not just produces English gibberish, but every now and then it can produce original coherent statements.

Again, I didn’t do this to produce a useful tool, but rather an exercise to learn how to build models.

They are all trained on copyrighted material without permission, no LLM is ethical

However, this means I can refute your statement because 100% of the input data is public domain novels. Also, I trained it on my own hardware powered 100% by solar power.

I’ve come round to the idea that I will not use LLMs regardless of how “ethically” they can be sourced because their outputs are harmful in a way that builds up, like a very small dose of poison.

How is LLM output any more harmful than output from a human you don’t know? I would agree with you if one were to simply blindly accept anything an LLM gives you as factual. However, I am skeptical of what humans say too. Some are inaccurate out of carelessness, some out of malice. Critical thinking is the key to protection from both errand LLM output and fallible humans.

Additionally, this thread is about locally running LLMs. One of the most dangerous aspects of most large LLMs is the sycophancy where the LLM will try to tell you what you want to hear, even if it needs to creatively invent things that don’t exist or are not true. If you are not aware, running locally means you have all the controls on the model. You’re not subject to whatever settings a large hyperscaler sets up for you. This means, among other things, you can turn down the “temperature” setting, which is the lever that controls how creative an LLM is. Practically what this means is, if you set it to “0” you are allowing it no creative action. If you ask it a question it doesn’t have actual training on, it will tell you effectively “I don’t know” instead of making something up just to have an answer.

It might seem like a romantic or artist’s approach, but with so much spiralling out of our control I’m trying to make the active choice to be more human.

The older I get the more disappointed I get in humanities large group decisions and actions.

Is it really feasible to run it 100% locally?

Yes. My Macbook Air (M2) released in 2022 can run many publicly available LLM models. The ouput is not as fast as using a large powerful datacenter, but for my local needs, I’m not in a hurry. I get about 17 to 30 tokens per second speed running 100% locally.

If yes to the previous: the software doesn’t come from nowhere

For Mac users, the interface comes from here . For the specific LLM models that is a separate question for each.

and ultimately still relies on gas-turbine-powered datacenters

DCs generating power on-site is a relatively new phenomon because existing grids are at capacity, so the only way to bring new DCs online is locally generating power at that DC, usually using gas turbines or even worse, diesel generators. Most if not all of the publicly available models for running on your own hardware were built before those gas-turnbine-generating DCs were a thing.

The public models people are running now have existed for a number of years are likely made on regular utility grid power which is whatever that nation and region uses.

and stolen IP and stolen personal data, no?

The Llama LLM is made by Meta, so probably yes for that one. Deepseek is from an AI research lab in China. QWEN is from Chinese company Alibaba. We don’t know for sure the inputs that created the Chinese models. US AI companies claim a number of the Chinese models are derived from American LLMs, but I haven’t seen (or looked for) proof of these claims.

If what I wrote above is true, what exactly are people arguing when they say it’s still possible to use LLMs ethically

Likely they mean because you don’t have to pay a large LLM owner in a rent-seeking model to run LLMs, nor does a person’s use contribute to further development by those companies.

or true to FOSS philosophy, because … ???

With the open-weight models it doesn’t rely on a commercial license to use, and the interfaces can be truly open source.

His response was ‘just do what you can’. I gave it a couple of months to see anything would come of our conversation but it didn’t.

This sounds like he was okay with you pushing back on any work that was too much and saying “no” to any task beyond your reasonable work limits. Even with this did you feel the damage to the working relationship was just too far gone?

Seems convoluted. Why not just let the business venture fail?

Usually because the business failure could mean thousands of primary job losses, and possible tens of thousands of secondary job losses from other businesses that support the first. Depending on the region, that could have massive ripple effects across the local, regional, and possibly national economy.

We don’t have a strong social safety net in the USA so that would likely lead to large scale poverty, homelessness, and massive increase food insecurity. These things then ripple out to other businesses in the area completely disconnected from the primary failing business causing more job losses as so forth.

The amount of public money to try to support the population and economically revitalize the region is massive when set next to a comparatively small bailout that would maintain the workforce at the primary business and and avoid the other businesses affected secondarily.

In short; its cheaper to pay the bailout so the primary company can pay their bills again than to operate in a pure market economy of boom and bust.

And if the basic function of the business is so integral to the function of society then it should not be a business, but should be a regulated administrered aspect of government.

If a business functioning is so important it needs to get bailed out, in this case it should be nationalized.

This can get messy very quickly:

From a domestic point of view, if the business is one of many providing the product or service, then the government now has a vested interest in driving any public spending to that one nationalized company while the surviving non-nationalized companies attempt to compete.

From an international point of view, if that nationalized company produces products for export, then you get the same problem we have with many companies in trade regulations. For example, the WTO prohibits direct subsidies tied to exports. The USA accuses China of this all the time and is the basis for many import bans in countries around the world.

Could be USA or Italy. What a sad world we live in.

Obviously the workload they were putting on you was way beyond a rational amount for one person. If you’re willing to talk about it, can I ask if you considered simply refusing to do more than “one person’s” amount of work and asked them to prioritize all your tasks so you could work on the things they deemed most important? Obviously this could lead to you being fired, but if you were already set to quit, the outcome would have been the same (except perhaps you’d be eligible for unemployment insurance).

Ah, so true: Russian Oligarchs.

Also i want to someday earn money without having a job

This is possible, but you need to have ownership of something with substantial value that others are willing to give you money for it. Unless you’re given something like this (usually inheriting it) the only way to get one of these is to buy it for a very large sum of money you get from something else that will allow you to earn a small fraction of what you paid for the item of value. So unless you inherent the item, no one starts with this, but its something you can get later in life if you get enough money from elsewhere.

or where i am my own boss

This is possible to do starting with nothing, but its a very hard path. Entrepreneurship pays way WAY less than most hourly jobs for years (possible decades) and you work way WAY harder than any hourly or salary job working for someone else. Finally, the risks to your income are far FAR higher than working for someone else. Failed businesses by entrepreneurs is extremely common, and you’re frequently left with less than nothing because its common to take on debt to start or grow a business. So your business could fail, but your debt doesn’t disappear when your business income disappears.

ive tried the “simple” ways to get a grasp of learning. Ive tried free methods like surveys, games, ebay selling, etc, but i find that those are a waste of time/effort for me atleast and im lucking to break even.

If you’re trying to succeed in areas where anyone can do the thing, you’re competing with everyone that can do that thing. That’s a low chance of succeeding, as you’ve discovered.

The winning move is to specialize. Shrink your competition by doing an activity that you, and a smaller number of people can do.

and i have no talent.

People typically categorize “talent” by specific job roles such as: music, computers, athletics. Instead of a whole career role, identify what individual traits or abilities you have. This is what career counseling is for.

• Are you very detail oriented and can follow a list of instructions perfectly time and time again?
• Do you have good anomaly detection where you can look at a result and realize something is wrong or missing?
• Are there any particular extreme conditions you are quite comfortable working long hours in (high temperature, small spaces, low or no light, etc).
• Do you have extensive cultural understanding of any particular groups (language, customs, traditions)?
• Do you know if you are good working for long periods alone or the opposite where you are constantly with others?
• Do you learn best by reading, hearing, or doing?

These kind of abstract questions, and your answers, can lead you to all kinds of paths that you could be particularly well suited for that others are not! This means you’d be competing with far fewer people for roles that match your specific traits.

In short, do you know yourself well enough yet? If not, seek all kinds of work or experiences that will help you answer questions like this. The most powerful thing for anyone to be successful, in however each of us defines that, is knowing who we each are. Only then do I believe we can each chart our personalized individual paths to success.

So for the average person who won’t get a multi-million dollar loan/inheritance from mommy and daddy

You don’t need a multi-million dollar loan/inheritance to retire with $5 million. You need substantially less and the time for it to compound. A person retiring today with nothing but boring a passive S&P500 investment of $50,000 and 40 years (average working years length) in 1985 would have $5,508,666 right now without ever making another contribution than the first $50k. That’s even going through all the financial downturns during that span of 1987 Black Monday stock market crash, Dotcom bubble burst, 9/11, Great Recession, and COVID.

I’m not disagreeing that most people don’t have $50k when they turn 18 to put into retirement, but I wanted to point out the power of the passive part of this that it only takes $50k with no other action but time to retire with over $5m.

Passive income doesn’t exist.

Roughly 5 million USD invested into stock market is enough to live a comfortable middle-class life off the interest.

…and still have the entire principle upon death. But you you still need $5m to exist when you die?

This means that unless you really have a need to pass on $5 million to your heirs, the number is closer to $2.5M-$3m and live the exact same lifestyle (same income from interest only) by gradually drawing down the principal while also taking the interest. Its impossible to know the year of our actual death (by natural causes), but you can likely get close and reduce the draw down amount to extend the money you have left then.

Even then spending in retirement isn’t usually linear anyway. Behaviorally spending will increase substantially when you initially retire for a time as you take your trips and indulge in your leisure activities in the first 5 or maybe 10 years, but your spending will gradually decline anyway as your health limits your activities for the next 10 to 20 years. Only really late in life as you get closer to death does your spending typically go back up (for medical expenses).

Because spending isn’t linear and you can drawdown your principal, the number declines again to about $1.5m-$2m needed to live a comfortable middle class lifestyle until death.

Sure, they are just worn under clothing because winter is cold.

I do remember a short time after Gardasil 9 came out that they were limiting it to young women because HPV is the cause of cervical cancer and the vaccine was in very short supply because large scale manufacturing hadn’t ramped up yet. I’m happy that time is over and its available to everyone.

i think you just gave me my first boomer memory moment. lol

No worries at all. All of us will have this at some point in our lives if we’re lucky enough to make it that far.

because american health insurance won’t cover it for men

Where does this idea come from? I’m a man and had no problem with insurance covering the Gardasil 9 (HPV vaccine) shots. A quick google search shows that vast majority of private health insurance plans as well as 90% of ACA plans cover it with little to no cost. Mine was 100% covered by insurance as a pharmaceutical without even a co-pay. I didn’t even need to go to my doctor and just when to the pharmacy who gave the shots (Costco in my case).

British humor.