Peel potatoes and cut into 1-inch chunks. In a large saucepan, cover potatoes with cold water by 2 inches and add 1 tablespoon coarse salt. Bring to a boil; cook until potatoes are very tender and easily pierced with a fork, 20 to 25 minutes. Drain; transfer to a large bowl.
Using a potato masher or fork, mash potatoes with olive oil and milk until smooth. Season with salt and pepper. Drizzle with more olive oil before serving, if desired.