Estabas referenciando el elemento de manera incorrecta. Probá así:
Código PHP:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<title>Documento sin título</title>
</head>
<script>
function ejecuta()
{
if(document.getElementById('VIDEO').getElementsByTagName('param')[0].getAttribute('value')=='d'){
document.getElementById('reproducir').disabled=true;
}
else{
document.getElementById('reproducir').disabled=false;
}
}
</script>
<body>
<table border="2" bgcolor="#ff0000">
<tr>
<td>
<object id="VIDEO" width="645" height="480"
style="position:center; left:0;top:0;"
classid="CLSID:6BF52A52-394A-11d3-B153-00C04F79FAA6"
type="application/x-oleobject">
<param name="URL" value="d">
<param name="SendPlayStateChangeEvents" value="True">
<param name="AutoStart" value="true">
<param name="uiMode" value="none">
<param name="PlayCount" value="9999">
<param name="enableContextMenu" value="">
<param name="Settings.balance" value="0">
</object>
</td>
</tr>
</table>
<form id="form1" name="form1" method="post" action="">
<input id="reproducir" type="button" name="Submit" value="Botón" onclick="ejecuta()" />
</form>
</body>
</html>