The idea would be they are more about getting resources in the wild but not strictly plants. You could generally add a carry weight increase and increase the number of logs/rocks/dirt the player can carry in their hand. Could also be renamed as a Laborer skill.
You could add prospecting with a crafting table that sifts for metal in river sand. Something like converting River sand into Regular sand with a 20% chance of a iron nugget, 5% chance of copper nugget and 0.2% of a gold nugget. Then you can sell the nuggets to a smelter who can turn them into a regular bar.
You could add recycling, have different trash types and each type can output raw materials when processed by a gathering table. So the gatherer can transition into recycling in the late game.
Being directly tied to plants forces you to either directly compete against or too heavily rely on farming. (You rely on them heavily as the natural plants can quickly disappear). It feels required that farmers take gathering, wheras a miner for example can go into masonry or glass making.