It is a good idea to begin learning kayak fishing with a used kayak. A used kayak costs considerably less than a new one. You can get a used one in good shape if you look around. Also important to deal with your local shop as they can best advise you on what is needed.
Search for one in the classifieds and in online auctions. Be sure to check on the size and shape of the kayak. It should suit your height, torso size, and paddling style. A paddle made of the aluminum shaft is best suited for a beginner. You can choose paddles made of wood, fiberglass or carbon fiber later when you become familiar with kayaking. You can buy kayak accessories online.
Kayaks are relatively cheap compared to other fishing vessels. A decent not too fancy kayak set up for fishing will usually run less than one thousand dollars. There are kayaks with pedal drive and all of the bells and whistles that may cost close to three thousand dollars but a basic fishing kayak should cost less than one thousand dollars.
The ability to be stealthy is a definite pro for this style of fishing. A trolling motor is not as quiet as everyone thinks. Fish can hear very well and often are spooked by the hum of a trolling motor. Kayaks don't have that problem and can easily sneak up on fish.