2. To make the citrus relish, combine the grapefruit and orange sections in a mixing bowl. Add the lime juice, ginger and cilantro and set aside. (This can be made in advance and stored in the refrigerator for up to 3 days. The relish should be served at room temperature.)
3. Meanwhile, drizzle the swordfish steaks with olive oil and season with salt and pepper. Grill or broil the fish on each side until it is cooked through, about 6 minutes per side.
4. Place each steak on a plate, spoon some of the citrus relish on top of each steak and serve.