In large mixing bowl, combine mashed potatoes with onions, celery tops, parsley, dill, paprika, nutmeg, and any optional add-ins.
Mix in egg, salt, and pepper. Fold in flaked fish.
Using your hands, portion out about 1/4 cup of mixture. Form it into fish cake about 2 to 3 inches in diameter and 1 inch thick and place on baking sheet. Repeat with rest of the mixture. If you have time, place in fridge for 20 minutes to allow them to firm up and hold their shape better during frying.
Preheat large cast-iron pan or sauté pan over medium heat. Add oil and fry cakes until well browned on both sides, 7 to 9minutes total.