glx: Add safe_{add,mul,pad} (v3) [CVE-2014-8093 4/6] (v4)

These are paranoid about integer overflow, and will return -1 if their
operation would overflow a (signed) integer or if either argument is
negative.

Note that RenderLarge requests are sized with a uint32_t so in principle
this could be sketchy there, but dix limits bigreqs to 128M so you
shouldn't ever notice, and honestly if you're sending more than 2G of
rendering commands you're already doing something very wrong.

v2: Use INT_MAX for consistency with the rest of the server (jcristau)
v3: Reject negative arguments (anholt)

v4: RHEL5: add limits.h, use inline

v5: backport to nx-libs 3.6.x (Mike DePaulo)

Reviewed-by: Keith Packard <keithp@keithp.com>
Reviewed-by: Julien Cristau <jcristau@debian.org>
Reviewed-by: Michal Srb <msrb@suse.com>
Reviewed-by: Andy Ritger <aritger@nvidia.com>
Signed-off-by: Adam Jackson <ajax@redhat.com>
Signed-off-by: Alan Coopersmith <alan.coopersmith@oracle.com>
Signed-off-by: Fedora X Ninjas <x@fedoraproject.org>
Signed-off-by: Dave Airlie <airlied@redhat.com>

1 files changed, 41 insertions, 0 deletions

diff --git a/nx-X11/programs/Xserver/GL/glx/glxserver.h b/nx-X11/programs/Xserver/GL/glx/glxserver.h
index e8449b2bb..404757405 100644
--- a/nx-X11/programs/Xserver/GL/glx/glxserver.h
+++ b/nx-X11/programs/Xserver/GL/glx/glxserver.h
@@ -54,6 +54,7 @@
 #include "GL/glx_ansic.h"
 
 
+#include <limits.h>
 /*
 ** The X header misc.h defines these math functions.
 */
@@ -223,6 +224,46 @@ extern void glxSwapQueryServerStringReply(ClientPtr client,
 /*
  * Routines for computing the size of variably-sized rendering commands.
  */
+static __inline__ int
+safe_add(int a, int b)
+{
+    if (a < 0 || b < 0)
+        return -1;
+
+    if (INT_MAX - a < b)
+        return -1;
+
+    return a + b;
+}
+
+static __inline__ int
+safe_mul(int a, int b)
+{
+    if (a < 0 || b < 0)
+        return -1;
+
+    if (a == 0 || b == 0)
+        return 0;
+
+    if (a > INT_MAX / b)
+        return -1;
+
+   return a * b;
+}
+
+static __inline__ int
+safe_pad(int a)
+{
+    int ret;
+
+    if (a < 0)
+        return -1;
+
+    if ((ret = safe_add(a, 3)) < 0)
+        return -1;
+
+    return ret & (GLuint)~3;
+}
 
 extern int __glXTypeSize(GLenum enm);
 extern int __glXImageSize(GLenum format, GLenum type,