Try changing the line:
    set JAVAOPTS=-Denv.windir=%WINDIR%
 
to
    set JAVAOPTS=-Xms256m -Xmx512m -Denv.windir=%WINDIR%
 
You may also need to put quotes around the %JAVAOPTS% in the last line, but it looks like you can get away without them.
 
If you still get heap errors, try:
    set JAVAOPTS="-Xms512m -Xmx1024m -Denv.windir=%WINDIR%"
 
By the way, if you're using 32-bit Vista, you only have available about 3.0 to 3.5 GB of RAM of the 4.0 GB installed, depending on your hardware configuration. This is not a Vista or Windows limitation: 32-bit versions of OS X and Linux are also limited to 3.0-3.5 GB of RAM on Intel machines with 4 GB installed. The reason why is a long story, so let's call it a hardware limitation.

From: oxygen-user-bounces@oxygenxml.com [mailto:oxygen-user-bounces@oxygenxml.com] On Behalf Of Mark Wilson
Sent: Thursday, March 05, 2009 12:13 PM
To: oxygen-user@oxygenxml.com
Subject: [oXygen-user] Out of memory problem

This is likely not the right list for this question, but please help if you can.

Using Oxygen's scenario mechanism, I am able to run the built-in FOP and produce my PDF file with no problems. However, when I try to do the same thing from a BAT using my stand-alone FOP 0.95, I get an "out of java heap space memory". I Googled and found lots of information for people who know java or Linux (I know neither). I know I have to change the fop.bat file but do not understand exactly how to do that. Concerning java, the fop.bat has the lines:

set JAVAOPTS=-Denv.windir=%WINDIR%

if "%JAVA_HOME%" == "" goto noJavaHome
if not exist "%JAVA_HOME%\bin\java.exe" goto noJavaHome
if "%JAVACMD%" == "" set JAVACMD=%JAVA_HOME%\bin\java
goto runFop

:noJavaHome
if "%JAVACMD%" == "" set JAVACMD=java

:runFop
rem ECHO "%JAVACMD%"
"%JAVACMD%" %JAVAOPTS% %LOGCHOICE% %LOGLEVEL% -cp "%LOCALCLASSPATH%" org.apache.fop.cli.Main %FOP_CMD_LINE_ARGS%
What do I add or change?

 I have 4 gig of memory on a Vista machine. What do I add or change so that my stand-alone FOP will have the same heap size as the Oxygen built-in FOP?
 
Thanks for any consideration you can give me,
Mark