We introduce a jet tagger based on a neural network analyzing the Minkowski functionals (MFs) of pixelated jet images. The MFs are geometric measures of binary images, and they can be regarded as a generalization of the particle multiplicity, which is an important quantity in jet tagging. Their changes by dilation encode the jet constituents’ geometric structures that appear at various angular scales. We explicitly show that this analysis using the MFs and dilation can be considered a constrained convolutional neural network (CNN). Conversely, CNN could model the MFs in the limit of a large network. We show an example that the CNN decision boundary correlates strongly with the value of MFs in semivisible jet tagging of a hidden valley scenario. The MFs are independent of the infrared and collinear (IRC)-safe observables commonly used in jet physics. We combine this morphological analysis with an IRC-safe relation network which models two-point energy correlations. While the resulting network uses constrained input parameters, it shows comparable dark jet and top jet tagging performances to the CNN. The architecture has significant computational advantages when the available data is limited. We show that its tagging performance is much better than that of the CNN with a small number of training samples. We also qualitatively discuss their parton shower model dependency. The results suggest that the MFs can be an efficient parametrization of the IRC-unsafe feature space of jets.