Cancer is not a single disease. There are over two hundred different types of cancer that can affect our bodies. Even within a single organ like the breast or the large bowel one of several different sorts of cancer may develop, each of which needs a slightly different approach to treatment.
Surgery is still the most widely used treatment for cancer. The hope is that when the surgeon operates to remove the growth they will be able to take it away completely and cure the problem.
When a cancer is removed in this way it is always examined under the microscope by a pathologist, who will also look at samples of the surrounding normal tissues and (sometimes) lymph glands that were taken away at the same time. This lets the doctors know exactly what type of cancer was present. It also gives an idea of whether it was removed completely or whether there is a risk that some cells might have been left behind or might have spread to other parts of the body.
If the pathologist's examination (and possibly other tests) suggest there is a risk that the cancer might have spread, even though there are no obvious signs of this, then chemotherapy may be recommended. Chemotherapy will only be suggested, however, if that particular type of cancer is known to be sensitive to, and benefit from, drug treatment.
The results of many clinical trials have shown that when there is a risk of a breast cancer having spread, giving chemotherapy after an operation can increase the likelihood of a cure. The same is also true for large bowel cancers when there is a risk they may have spread. So, for example, if your own growth had been a very early tumour with very little risk of spread then there would be little or no benefit from having chemotherapy.
The real message is that every person and every cancer is different and treatment is carefully matched to the individual situation. The best thing, if you still have any concerns, is to discuss this question with your own specialists and see exactly why they felt that chemotherapy was not needed in your case.

