You know your family best, but I would definitely consider what your vet said very carefully. Keep in mind that it is not in his best interests financially to tell you to return your dog as he won't be getting any more of your business if you decide not to get another dog. So if he's suggesting that you should think about returning the puppy, it is more than likely that his motivation is that he's genuinely concerned for the safety of this dog.
Toy breeds and puppies can certainly be injured by accidents or inexperienced handling. A toy breed puppy combines the risks of a small breed and a young dog. I think your vet's concerns are legitimate.
How old are your children? How capable are they of understanding that this small puppy is not an actual toy but a living creature with feelings than can be hurt if not treated well? Do your kids listen to you when you tell them to settle down or leave something or someone alone? Are there any toddlers in your family who are less than steady on their feet and capable of falling over on your puppy? Will your kids undermine your efforts to socialize or train your puppy - intentionally or not - by dangling food in front of the puppy, feeding it under the table, teasing it, or chasing it around?
In general, puppies and very young children do not mix all that well. I'm sure their are exceptions, but in general, neither the puppy nor the children understand proper boundaries for interactions and one or the other is likely to get hurt because of it. Puppies also require a lot of work; it's not unlike having another young child. A larger, adult dog who has experience being around children may be a better fit for your family. Or if you are set on getting a puppy, you could wait until your children are old enough to understand how to behave around a puppy.
At the very least, I would discuss your vet concerns with the breeder and possibly a professional dog trainer. Only you can make this decision, so consider carefully what's going to be best for the dog and your family.
Edit: Regarding your additional information, I'm very concerned to hear that the breeder doesn't seem to be interested in doing anything about your concerns. At the very least they should be giving you convincing arguments as to why they think a toy puppy will be OK in your home. Most reputable breeders will take any puppy they have sold back at any time for any reason, including far less legit ones than yours. Is there anything in the contract about what you can or must do if you decide you don't want the dog?
Knowing that you already have dogs does help the situation. Hopefully that means you've had adequate time and opportunity to teach your kids how to treat dogs gently and with respect. But you mention that they're big dogs and their certainly is a difference. One of your kids could lean on or even sit on one of your big dogs without any consequences. If your dogs are good with the kids and well trained, the worst thing the dog will do is get up and walk away. But of course, the exact same behavior could be disasterous for both child and dog when dealing with a toy breed puppy.