By default, each channel shows the ratio of the input to the output, In / Out. This is useful for measuring the transfer function of a device under test. The math channel allows you to plot different combinations of Ch 1 and Ch 2. If the output amplitudes of both channels are set to the same value, then viewing the math channel as Ch 1 / Ch 2 will show the ratio In 1 / In 2, since the outputs are the same.
The In is referencing the input to the Moku, which would be the output of the DUT. The Out is referencing the output of the Moku, which is a swept sine wave and the input to the DUT.