There really isn't a good "natural" way to get RID of canker sores. There are natural ways to help soothe them though. The first thing to remember (which shouldn't come as a surprise) is to eat foods and drink liquids with as low of an acid content as possible. Acid will only irritate the sores and make them feel worse.
Secondly, and only try this if you have a strong stomach, is to take a bit of clove oil, preferably on a Q-Tip, and rub that on the sores. That will completely eliminate the pain for a few hours. I say "a strong stomach" because the flavor is extremely strong and well, yucky.
On a side note, a not so natural treatment that is great for treating canker sores, and just about anything else that is inside your mouth (such as if you bite your tongue, or if you get a piercing, etc) is a liquid called Gly-Oxide. It comes in a little white bottle and is sold in the drugs/pharmacy section of most stores, including Wal-Mart. It also kills all the germs in your mouth, disinfects, and soothes. It has a very mild agreeable flavor and works like gold!