Ограничение использования оперативной памяти программы

Вы могли добавить область подкачки, читать man mkswap;man swapon чтобы заставить систему думать, это имеет больше памяти.

Для ограничения использования оперативной памяти читать man bash и поиск ulimit:

   ulimit [-HSabcdefiklmnpqrstuvxPT [limit]]
          Provides control over the resources available to the shell and to processes started
          by  it, on systems that allow such control.  The -H and -S options specify that the
          hard or soft limit is set for the given resource.  A hard limit cannot be increased
          by a non-root user once it is set; a soft limit may be increased up to the value of
          the hard limit.  If neither -H nor -S is specified, both the soft and  hard  limits
          are set.  The value of limit can be a number in the unit specified for the resource
          or one of the special values hard, soft, or unlimited, which stand for the  current
          hard  limit,  the  current  soft  limit,  and  no limit, respectively.  If limit is
          omitted, the current value of the soft limit of the resource is printed, unless the
          -H  option  is given.  When more than one resource is specified, the limit name and
          unit are printed before the value.  Other options are interpreted as follows:
 ...
          -d     The maximum size of a process's data segment
 ...
          -l     The maximum size that may be locked into memory
 ...
         -v     The  maximum  amount  of  virtual memory available to the shell and, on some
                 systems, to its children