Here's an article showing how you can limit the number of rows in a multi row variable set. Unfortunately there is no simple setting for this and you have to script this.

1. Create an  onLoad  client script on the catalog item (or record producer) level:

   function onLoad() {
   GLOBAL_VAR = {
   g_form: g_form
   };
   }

1. Create an  onSubmit  client script at the variable set level:

   function onSubmit() {

   var numberOfAllowedRows = 2;//Pick depending on your validation
   var mrvs = GLOBAL_VAR.g_form.getValue('your_variable_set_name').toString();
   if (mrvs) {
       if (JSON.parse(mrvs).length == numberOfAllowedRows) {
           alert('Only ' + numberOfAllowedRows + ' rows allowed');
           return false;
       }
   }
   

   }

Note :

The variable set name you will need to use in the second script is the  internal name  of the variable set.

Set the  UI Type  to  All  on both the client scripts so they run on portal.

Conclusion:

This is a slightly more elegant way compared to simply writing an onSubmit script at the catalog item (or record producer) level and counting the rows. Using a combination of the above 2 scripts the user is pre-informed of the number of rows he/she can submit. Of course, it would be really cool to perform the validation on click of the Add button, but you don't want to mess with DOM, do you!?