unstable: never settled or at rest, like something that cannot be held still
The word pictures something that will not stay put. It is built on a verb meaning to set, place, or establish, with a prefix that negates it, so the literal sense is "not set in place, not settled." Both of its New Testament appearances are in the letter of James, and both keep this one idea while applying it to different things.
In James 1:8 it describes a person: the double-minded man is unstable in everything he does. In James 3:8 it describes the tongue: a restless evil that no one can tame. The thread running through both is the same picture of something that cannot be brought to rest.
Behind the personal sense lies a wider and more violent range that the word carries elsewhere in Greek and in the matching noun. The same root, in its noun form, names public upheaval: riots, revolts, the chaos of a society coming apart. James seems to draw on that resonance. A divided heart is not just a private wobble; left alone it churns out the kind of disorder that tears a community apart, which is exactly what the matching noun describes later in his letter.