But there's another important point to consider before you go too far down the collecting path, which may actually determine the subject and focus of your collection.
Have a plan for the organization and storage/display of your collection
Let's break that principle down into its component parts.
Storage - It's usually not the first thing a collector thinks of, but it's often the first problem one runs into. Say your interest is nautical memorabilia -- and you live in a tiny New York City apartment. Deciding to collect ship's wheels wouldn't be very practical. But steamship dinner menus, small box compasses, a sextant or two, etc. could be things that one could collect and enjoy in such a small space.
And storage isn't just a question for bulkier items, like oil paintings or Chippendale furniture. Even small objects can reach critical mass.
I know someone who has a massive book collection. Now she doesn't follow our second principle -- have a focus. She continually brings home boxes of books from auctions and estate sales that are only marginally related to her primary subject, and after years of doing this she has yet to separate the wheat from the chaff of her collection.
Her library has outgrown the house, as well as the small detached building built for her use. A portion of the barn on the property has also been converted to book storage, and that's currently overflowing as well.
Here's the thing -- many of those books which she "rescued" from the dumpster by bringing them home are deteriorating a rapid rate. The volumes in the barn are exposed to extreme temperature, excessive moisture, and various critters who find old paper quite tasty.
Had she kept the confines of her library building, her collection of books might be one of a limited number of volumes, but one that could be continually refined and upgraded. As it is, most of her collection is simply disintegrating because storage wasn't considered.
Collect your passion, but consider the space you have available to indulge it.
Organization - So let's assume you have the space you need for your collection to grow into. How will you organize it? Thinking about this may also help the direction your collection will take. Decided to collect stamps? Cool. Chances are the albums you purchase will help you organize your treasures better than a shoe box.
Collect LPs? Then organize them by label, year, genre, artist, or some other theme that makes sense to you. Dolls can be arranged by size, or perhaps by age. Vintage stock certificates by company, or engraver, or year.
The point of organizing is two-fold. First, it's a good way to maintain an idea of what's in your collection. Because there will come a time (sooner or later), when you can't remember everything you have, and you'll start unintentionally duplicating objects.
Secondly, it's a good way to evaluate your collection. If you're looking to collect a particular run or sequence of objects (like all the baseball cards of the 1954 American League), then you can readily find the gaps and know what you should look for.
It's also an opportunity to take a hard look at what you have and ask what objects need to remain. Organizing isn't a one-time activity. Regular organizing helps you reevaluate your collection and help you keep it pruned. After all, tastes change, and sometimes those oh-so-desirable objects have become "what-was-I-thinking?" embarrassments. Let your collection change and grow with your interests.
Display - And finally, there's the question of display. Perhaps you don't want to show off your collection. That's your choice, of course. Maybe for security reasons you'd rather not have everyone know just what stamps or coins you've collected.
But since a good collection reflects the collector's passion, it's safe to assume that you'd want to display some of it. Maybe your collection is such that only a few pieces can be shown. You may have thousands of antique post cards, for example, but only choose (or have the space) to frame and hang ten of the most interesting.
Maybe you'll want to have a bookcase for your collection, or (depending on the size) a dedicated room. While your collection may start small, it's a good idea to think of these kinds of things early.
I know someone who has their collection scattered about their home. Things were just placed anywhere temporarily -- and never moved. Just about every flat surface is filled with something from their collection, which makes it hard to see exactly what the collection's about (and kind of looks like the first stage of hoarding to me).
So go ahead and collect those soda pop bottle caps. But while you still just have five or six, think about what you're going to do if you accumulate several hundred of them. Having piles of bottle caps on a card table is not an attractive long-term solution.