That would likely be the S11 magnitude - in the passband, the filter
lets the RF through, so it sees the 50 ohms on the other side, so
reflected power is small (i.e. good match). Outside the passband, the
filter probably reflects the power back, so S11 is close to 1.

S21 should show the bandpass characteristic (i.e. small magnitude (large
negative dB) outside the passband and big magnitude (small negative dB))
inside the passband.

Do you set the frequency range say from 20 Mhz to 50 Mhz, and set the centre frequency to 30 Mhz, and then sweep the frequency to get the graph?
What about SPAN and RBW (band width) - what do you set it to?
And it is set to read LOGMAG of S21?
What is supposed to happen on S11?