We study the operator algebra of extended conformal defects in more than two spacetime dimensions. Such algebra structure encodes the combined effect of multiple impurities on physical observables at long distances as well as the interactions among the impurities. These features are formalized by a fusion product which we define for a pair of defects, after isolating divergences that capture the effective potential between the defects, which generalizes the usual Casimir energy. We discuss general properties of the corresponding fusion algebra and contrast with the more familiar cases that involve topological defects. We also describe the relation to a different defect setup in the shape of a wedge. We provide explicit examples to illustrate these properties using line defects and interfaces in the Wilson-Fisher CFT and the Gross-Neveu(-Yukawa) CFT and determine the defect fusion data thereof.