In a large mixing bowl, combine the first 9 ingredients. Mix in egg; then mix in flours until thoroughly combined.
In a 10-inch nonstick skillet over medium heat, heat oil.
Working in batches of 3 to 4 fritters at a time, drop 1 tablespoon batter per fritter into the oil and cook until golden brown, about 1 minute each side.
Transfer fritters to a paper-towel-lined plate to drain; season with a pinch of salt and a squeeze of lime.
Serve immediately with remaining lime wedges and, If you like, store-bought pepper jelly ( we use the Stone Wall Kitchen Red Pepper Jelly) and/ or sour cream