path: root/nx-X11/programs/Xserver/hw/dmx/doc/scaled.txt

  Scaled Window Support in DMX
  Rickard E. Faith and Kevin E. Martin
  15 October 2003 (created 19 September 2003)

  This document investigates the possibility of adding scaled window
  support to the DMX X server, thereby allowing a window or some
  selected part of the logical DMX area to be displayed using a scaling
  factor.  For example, this might allow the contents of a window to be
  magnified for easier viewing.  In particular, scaling for the VNC
  client is explored.  _C_o_p_y_r_i_g_h_t _2_0_0_3 _b_y _R_e_d _H_a_t_, _I_n_c_._, _R_a_l_e_i_g_h_, _N_o_r_t_h
  _C_a_r_o_l_i_n_a

  ______________________________________________________________________

  Table of Contents


  1. Introduction
     1.1 DMX
     1.2 Problem Statement
     1.3 Task

  2. Previous Work
     2.1 VNC
        2.1.1 Scaling under VNC
     2.2 The X Video Extension

  3. Possible Solutions
     3.1 VNC-like Scaling
        3.1.1 Software Scaling
        3.1.2 Scaling with the X Video Extension
           3.1.2.1 Implementing the X Video Extension for DMX
           3.1.2.2 Supporting RGB formats for the X Video Extension
        3.1.3 Scaling with an XPutImageScaled Extension
        3.1.4 Scaling with an XCopyAreaScaled Extension
        3.1.5 Scaling with OpenGL
     3.2 Application-transparent Scaling for DMX
        3.2.1 Back-end Scaling Without Disconnect/Reconnect
        3.2.2 Back-end Scaling With Disconnect/Reconnect
        3.2.3 Server-side Scaling
     3.3 XCreateScaledWindow API
        3.3.1 XCreateWindow
        3.3.2 XSetWindowAttributes
        3.3.3 XGetWindowAttributes, XGetGeometry
        3.3.4 Popup and Child window positions
        3.3.5 Events
        3.3.6 Implementation

  4. Conclusion and Recommendations


  ______________________________________________________________________

  11..  IInnttrroodduuccttiioonn

  11..11..  DDMMXX

  The DMX X server (Xdmx) is a proxy server that is designed to allow X
  servers on multiple machines to be combined into a single multi-headed
  X server.  Combined with Xinerama, these heads can appear as a single
  very high-resolution screen.  Typical applications include the
  creation of a video wall with 16 1280x1024 displays arranged in a
  rectangle, for a total resolution of of 5120x4096.



  11..22..  PPrroobblleemm SSttaatteemmeenntt

  Applications displayed on a physically large video wall that provides
  high pixel-resolution may be difficult to see, especially if the
  application is designed for use on a typical desktop computer with a
  relatively small display located close to the human operator.  The
  goal of this paper is to describe and discuss solutions to this
  problem.

  The original driving problem for this work is to provide scaling for
  the vncviewer application when displayed using DMX (VNC scaling is
  currently available only with the Windows client, and there is no plan
  to extend that capability to other clients).  While this specific
  problem will be addressed in this paper, the general solution space
  will also be explored, since this may lead to a good solution not only
  for vncviewer but also for other applications.

  11..33..  TTaasskk

  For reference, here is the original description of the task this paper
  addresses:

  +o  Scaled window support (for VNC)

     +o  Investigate possibility of implementing a "scaled window"
        extension:

        +o  Add XCreateScaledWindow call that could be used in place of
           XCreateWindow

        +o  All primitives drawn to scaled window would be scaled by
           appropriate (integral?) scaling factor

     +o  Alternate approach: special case VNC support

  22..  PPrreevviioouuss WWoorrkk

  This section reviews relevant previous work.

  22..11..  VVNNCC

  22..11..11..  SSccaalliinngg uunnddeerr VVNNCC

  When using the vncviewer program for Windows, it is possible to
  specify a scaling factor (as numerator and denominator).  When scaling
  is in effect, the viewer software uses StretchBlt (instead of BitBlt)
  to display the pixels for the user.  When this call is made, the
  viewer already has received all of the pixel information (at full
  unscaled resolution).

  The scaling in VNC is primitive.  It does not conserve bandwidth, it
  does not treat textual information differently (i.e., by using a
  suitably scaled font), and it does not provide any anti-aliasing other
  than that provided by the underlying (Windows-only) system library.

  22..22..  TThhee XX VViiddeeoo EExxtteennssiioonn

  The X Video Extension is a widely-available extension to the X11
  protocol that provides support for streaming video.  Integral to this
  support is the ability to arbitrarily scale the output.  In version
  2.2 of the X Video specification, support for scaled still images was
  provided, using both shared memory and traditional transport.  The API
  for this support uses calls that are quite similar to XCreateWindow,
  XPutImage, and XShmPutImage.  Currently, most of the drivers
  implemented in XFree86 only support data in various YUV formats.
  However, several modern video adaptors support RGB as well.
  Note, though, that the target output for this scaling is an overlay
  plane -- so X Video provides functionality that is fundamentally
  different from that provided by the Windows StrechBlt call.

  33..  PPoossssiibbllee SSoolluuttiioonnss

  This section briefly discusses possible solutions, including major
  advantages and disadvantages from both the implementation and the end-
  user programmer standpoint.

  33..11..  VVNNCC--lliikkee SSccaalliinngg

  33..11..11..  SSooffttwwaarree SSccaalliinngg

  The vncviewer application could be modified to provide software
  scaling.  This is not a general solution, but it does solve one of the
  goals of this work.

  A prototype of this solution was implemented and a patch against
  vnc-3.3.7-unixsrc is available in the dmx/external directory.  Because
  of limited time available for this work, all of the edge cases were
  not considered and the solution works well mainly for integer scaling.

  Currently, vncviewer writes to the X display with XPutImage,
  XCopyArea, and XFillRectangle.  All instances of these calls have to
  be aware of scaling and must round correctly.  In the prototype
  solution, rounding is incorrect and can cause artifacts.

  A better solution would be to cache all updates to the desktop image
  in vncviewer and only send the damaged area to the X display with
  XPutImage.  This would allow the damaged area to be computed so that
  rounding errors do not create artifacts.  This method is probably
  similar to what is used in the Window client.  (The whole VNC suite is
  being re-written in C++ and the forthcoming version 4 has not been
  evaluated.)

  33..11..22..  SSccaalliinngg wwiitthh tthhee XX VViiddeeoo EExxtteennssiioonn

  The scaling in the Windows vncviewer application makes use of a scaled
  blit that is supplied by the underlying system library.  Several video
  cards currently provide support for a scaled blit, and some X servers
  (including XFree86) expose this capability to applications via the
  XvPutImage interface of the X Video Extension.  The capability exposed
  by XvPutImage results in the scaled image being drawn to an overlay
  plane.  Most video cards also provide support for a scaled blit into
  the normal output planes, but this is not exposed via XvPutImage.

  The vncviewer program could be modified to use the X Video Extension
  to provide scaling under X11 that is similar to the scaling currently
  provided under Windows.  Unfortunately, Xdmx does not currently export
  the X Video Extension, so this would not provide an immediate solution
  usable with DMX.

  A very early-stage proof-of-concept prototype was implemented and a
  preliminary patch against vnc-3.3.7-unixsrc is available in the
  dmx/external directory.  This prototype was implemented to better
  understand the problems that must be solved to make this solution
  viable:

  +o  As noted under the software scaling section above, vncviewer writes
     to the X display with several different calls.  These calls write
     to the normal output planes and are compatible with XvPutImage,
     which writes to an overlay plane.  To eliminate artifacts caused by
     this problem, vncviewer should be modified so that a cached copy of
     the desktop is available, either as a client-side image or a
     server-side off-screen pixmap, so that XvPutImage would be the only
     method for writing to the X display.

  +o

     Although several modern graphics adaptors support hardware scaling
     using an RGB format (e.g., ATI Radeon, nVidia, etc.), XFree86
     drivers typically only implement YUV formats.  YUV generally
     compress the pixel information in some way.  For example, two
     commonly implemented formats, YUY2 and UYVY provide intensity
     information for every RGB pixel, but only provide chroma and
     luminance information for pairs of horizontal pixels.  Since VNC
     uses pixel-resolution for communicating updates on the wire,
     additional artifacts are introduced (because there may not be
     enough information from the wire to update a pair of pixels).

     Further, the well-known problem with YUV encoding is even more
     evident when the image is a desktop instead of a movie.  For
     example, consider a 1-pixel-wide vertical window border.  If the
     border changes in color but not intensity (e.g., because a window
     manager uses color to indicate focus), there may or may not be a
     change in the YUY2 image, depending on the algorithm used for RGB
     to YUV conversion and on how the border pixel is ordered in the
     pair of pixels used by the algorithm.

     Many of these artifacts could be eliminated if vncviewer cached a
     complete RGB image of the desktop, and only did the conversion to
     YUV for properly aligned areas of damage.  The remaining artifacts
     could be eliminated if an RGB format was used with X Video (which
     may require the extension of existing XFree86 drivers to support
     RGB).

  +o  Most modern video cards support exactly one overlay plane that is
     suitable for use with X Video.  Therefore, only one application can
     use X Video at any given time.  This is a severe limitation in a
     desktop environment.

  33..11..22..11..  IImmpplleemmeennttiinngg tthhee XX VViiddeeoo EExxtteennssiioonn ffoorr DDMMXX

  The user-level API for X Video is fairly simple, but the underlying
  support required for the full specification is large.  However, since
  the API provides a method to query supported capabilities, a usable
  subset of X Video can be implemented that would support XvPutImage and
  little else.  This would require support for the following:

  +o  X Video Extension API calls, including the following:

     +o  XvQueryExtension

     +o  XvQueryAdaptors

     +o  XvQueryPortAttributes

     +o  XvFreeAdaptorInfo

     +o  XvListImageFormats

     +o  XvGrabPort

     +o  XvCreateImage

     +o  XvPutImage

     +o  XvShmCreateImage

     +o  XvShmPutImage

  +o  Support for querying back-end X Video Extension capabilities.

  +o  Support for sending the image to the back-ends.  Because X Video
     requires sending full images, there may be a trade-off between
     bandwidth limitations and additional complexity to divide the image
     up such that is scales properly.

  +o  Possible support for a software fall-back.  For example, if all of
     the back-ends do not support the X Video Extension, software
     scaling can be implemented such that the image is sent to the back-
     end with XPutImage.  This pathway would have poor performance.

  33..11..22..22..  SSuuppppoorrttiinngg RRGGBB ffoorrmmaattss ffoorr tthhee XX VViiddeeoo EExxtteennssiioonn

  Assuming an XFree86 driver already supports the X Video Extension, and
  assuming the target hardware supports an RGB format, then adding
  support for that format is relatively simple and straightforward.

  33..11..33..  SSccaalliinngg wwiitthh aann XXPPuuttIImmaaggeeSSccaalleedd EExxtteennssiioonn

  Instead of (or in addition to) implementing the X Video Extension in
  DMX, one obvious solution would be to implement a new extension that
  provides access to hardware-assisted scaled blits, similar to the
  StretchBlt call available under Windows.  This call would scale RGB
  images and would not use the overlay plane (unlike the X Video
  Extension).

  This approach has many of the same advantages and disadvantages as the
  XCopyAreaScaled Extension, discussed in the next section.  Discussion
  of XPutImageScaled is deferred in favor of XCopyAreaScaled for the
  following reasons:

  +o  XPutImageScaled can be emulated with XCopyAreaScaled by first using
     XPutImage to copy the image to an off-screen pixmap, and then
     calling XCopyAreaScaled between that off-screen pixmap and the
     target drawable.

  +o  Since XCopyAreaScaled would copy between two areas of on-screen or
     off-screen memory, it has additional uses and can be viewed as
     efficiently providing a superset of XPutImageScaled functionality.

  33..11..44..  SSccaalliinngg wwiitthh aann XXCCooppyyAArreeaaSSccaalleedd EExxtteennssiioonn

  As noted in the previous section, because XCopyAreaScaled provides a
  superset of the functionality provided by XPutImageScaled, we will
  consider this extension instead.

  First, XCopyAreaScaled would provide for RGB scaling between pixmaps
  (i.e., on-screen or off-screen areas of memory that reside on the
  video card).  Unlike the X Video Extension, which writes into an
  overlay plane, XCopyAreaScaled would write into the non-overlay areas
  of the screen.  Key points to consider are as follows:

  +o  Because different planes are involved, the two scaling operations
     are usually implemented in hardware differently, so an
     XCopyAreaScaled extension could be added in a manner that would
     neither conflict with nor interact with the X Video extension in
     any way.

  +o  The XCopyAreaScaled extension provides new functionality that the X
     Video Extension does not provide.  Based on anecdotal feedback, we
     believe that many people outside the DMX and VNC communities would
     be excited about this extension.

  +o  The main drawback to this extension is that it is new and needs to
     be implemented at the driver level in XFree86 for each video card
     to be supported.  At the present time, it is more likely that the X
     Video Extension will be implemented for a particular piece hardware
     because the X Video extension has multimedia uses.  However, over
     time, we would expect the XCopyAreaScaled extension to be
     implemented along with the X Video extension, especially if it
     becomes popular.

  +o  Another drawback is that not all modern cards provide support for a
     simple scaled blit operation.  However, these cards usually do
     provide a 3D pipeline which could be used to provide this
     functionality in a manner that is transparent to the client
     application that is using the XCopyAreaScaled extension.  However,
     this implementation pathway would make this extension somewhat more
     difficult to implement on certain cards.

  33..11..55..  SSccaalliinngg wwiitthh OOppeennGGLL

  Another general solution to the scaling problem is to use the texture
  scaling found in all 3D hardware.  This ability is already exposed
  through OpenGL and can be exploited by clients without X server
  modification (i.e., other than the ability to support OpenGL).  An
  application using OpenGL would transmit the non-scaled image to the X
  server as a texture, and would then display a single non-transformed
  rect using that texture.  This also works around the single overlay
  problem with the X Video Extension as well as the need to implement
  additional scaled primitive extensions.

  The downside is that most OpenGL implementations require power of 2
  texture sizes and this can be very wasteful of memory if, for example,
  the application needs to scale a 1025x1025 image, which would require
  a 2048x2048 texture area (even a 640x480 image would require a
  1024x512 texture).  Another downside is that some OpenGL
  implementations have a limited about of texture memory and cannot
  handle textures that are very large.  For example, they might limit
  the texture size to 1024x1024.

  33..22..  AApppplliiccaattiioonn--ttrraannssppaarreenntt SSccaalliinngg ffoorr DDMMXX

  33..22..11..  BBaacckk--eenndd SSccaalliinngg WWiitthhoouutt DDiissccoonnnneecctt//RReeccoonnnneecctt

  VNC does scaling on the client side (in the vncviewer application).
  Implementing a similar solution for DMX would require support in the
  back-end X servers and, therefore, is not a general solution.

  XFree86 already implements some support for "scaling" that could be
  used with DMX: if, in the XF86Config file, multiple Modes are listed
  in the Display Subsection of the Screen Section, then pressing Ctrl-
  Alt-Plus and Ctrl-Alt-Minus can be used to iterate through the listed
  modes.  The display dimensions will change to the dimensions in the
  Modes line, but the logical dimensions of the X server (i.e., the
  dimensions that Xdmx knows about) will not change.

  Further, the dimensions of the XFree86 display are under software
  control (via the XFree86-VidModeExtension), so the Xdmx server could
  change the screen dimensions on a per-display basis, thereby scaling
  the information on part of that display.

  However, this scaling appears to have limited use.  For example,
  assume a 4 by 4 display wall consisting of 16 1280x1024 displays.  If
  all of the back-end servers were simultaneously configured to display
  640x480, the left hand corner of each display would be magnified, but
  the composite result would be unreadable.  Magnifying one display at a
  time could be usable, but could have limited utility, since the result
  would still be no larger than a single display.


  33..22..22..  BBaacckk--eenndd SSccaalliinngg WWiitthh DDiissccoonnnneecctt//RReeccoonnnneecctt

  Disconnect and reconnect features are not currently supported in DMX,
  but are scheduled to be implemented in the future.  These features,
  combined with the XFree86-VidModeExtension Extension, would allow an
  application to do the following:

  +o  Disconnect a specific back-end server (via the DMX Extension),

  +o  reconfigure the XFree86 back-end server resolution, and

  +o  reconnect the back-end server to DMX -- at a new origin with the
     new screen resolution.

  For example, consider a display wall consisting of 16 1280x1024
  displays with a total resolution of 5120x4096.  All of the screens
  could be disconnected, repositioned, and reconnected each at a
  resolution of 640x480.  The total resolution of the display wall would
  be 2560x1920, allowing a view of a selected area approximately one-
  fourth of the size of the DMX display.  This change would be
  completely application independent (except, perhaps, for a DMX-aware
  window manager).  When work at the increased resolution was completed,
  the back-end servers could be disconnected, reconfigured, and
  reconnected for the original 5120x4096 view.

  Support for this type of scaling can be implemented in a DMX-aware X11
  client assuming the DMX server support arbitrary disconnect and
  reconnect semantics.  Because this application cannot be written
  before disconnect/reconnect is implemented, this solution will not be
  discussed further in this paper.

  33..22..33..  SSeerrvveerr--ssiiddee SSccaalliinngg

  In earlier versions of DMX, a frame buffer was maintained on the
  server side, and XPutImage was used to move the information from the
  server to the client (similar to some early VNC implementations).  The
  use of a server-side frame buffer would allow the server to do
  scaling, but is not a recommended solution because of overall
  performance issues and server-side memory issues (i.e., the frame
  buffer would be very large for large display walls).

  Exploration of this path is not recommended.

  33..33..  XXCCrreeaatteeSSccaalleeddWWiinnddooww AAPPII

  The implementation of X Video Extension in DMX, and the use of
  XvPutImage by applications requiring scaling requires significant
  changes in DMX Further, XvPutImage is, essentially a scaled blit, and
  it is only useful for applications which are already using (or can be
  modified to use) XPutImage.  Therefore, a more general API will be
  discussed as another possibility.

  X applications typically create windows with the XCreateWindow call.
  A new extension could provide an XCreateScaledWindow call that could
  be used in place of the XCreateWindow call and be otherwise
  transparent to the application.  This would allow applications, even
  those that do not depend on XPutImage, to take advantage of window
  scaling.  In this section we describe how the call would work, what
  transparency it provides, and how to solve the potential problems that
  transparency creates.

  33..33..11..  XXCCrreeaatteeWWiinnddooww

  The XCreateWindow call takes width and height as parameters.  An
  XCreateScaledWindow call could take all the same parameters, with the
  addition of a scaling factor.
  33..33..22..  XXSSeettWWiinnddoowwAAttttrriibbuutteess

  An X11 window has several attributes that would have to be scaled:

  +o  Background and border pixmaps

  +o  Border width

  +o  Cursor

  33..33..33..  XXGGeettWWiinnddoowwAAttttrriibbuutteess,, XXGGeettGGeeoommeettrryy

  For transparency, calls that query the window attributes should return
  unscaled information.  This suggests that all unscaled pixmaps and
  window attributes should be cached.

  Unfortunately, a window manager requires the scaled geometry to
  properly decorate the window.  The X server can probably determine
  which client is acting as the window manager (e.g., because that
  client will select events that are used exclusively by the window
  manager).  However, other Scaled Window Extension aware clients may
  also need to determine the scaled geometry.  Therefore, at least two
  additional extension calls should be implemented:
  XGetScaledWindowAttributes and XGetScaledGeometry.

  33..33..44..  PPooppuupp aanndd CChhiilldd wwiinnddooww ppoossiittiioonnss

  Some applications may position popup and child windows based on an
  unscaled notion of the main window geometry.  In this case, additional
  modifications to the client would be required.

  33..33..55..  EEvveennttss

  Most events (e.g., for mouse motion) return information about the
  coordinates at which the even occurred.  These coordinates would have
  to be modified so that unscaled values were presented to the client.

  33..33..66..  IImmpplleemmeennttaattiioonn

  There are many implementation issues, some of which are similar to the
  issues involved in implementing the X Video Extension for DMX.  The
  window contents must be scaled, either by performing all operations to
  a frame buffer and then writing the image to the display (perhaps
  using hardware scaling support), or by modifying all of the various
  drawing operations to perform scaling.  Because of the complexity
  involved, the frame buffer option is recommended.

  44..  CCoonncclluussiioonn aanndd RReeccoommmmeennddaattiioonnss

  We recommend a three phase implementation strategy, based on how an
  application could be written to take advantage of scaling:

  1.

     The XCopyAreaScaled extension should be implemented, since this is
     the ideal solution for applications like VNC, and since making use
     of this extension will require minimal changes to applications that
     already use XPutImage or XCopyArea.

     The initial implementation work would include the design of the X
     protocol extension, writing this up in the usual format for
     extension documentation, implementation of the protocol transport
     pieces in XFree86, implementation of a software fall-back in
     XFree86 and DMX, one example hardware implementation for XFree86,
     and implementation of support for this extension in DMX.

     We suggest implementing the extension first on the ATI Radeon
     cards.  However, since these cards do not provide a 2D scaled blit
     primitive, the implementation would have to make use of the 3D
     texture engine to emulate a scaled blit.  This is recommended,
     since other modern graphics cards also do not provide a simple 2D
     scaled blit operation and an example of the more difficult
     implementation pathway would be helpful to others.

  2.

     Until XCopyAreaScaled is widely supported, applications that
     require scaling will have to fall back to another scaling method.
     We suggest OpenGL as the first fall-back method because it is
     widely available and supported by DMX.

     A project centered around OpenGL-based scaling would implement this
     scaling in VNC as an example.  This work would include re-writing
     the vncviewer rendering engine to cache a master copy of the
     desktop image for all operations.

  3.

     Since OpenGL is not implemented everywhere, and may not provide
     hardware-assisted performance in every implementation, an
     application that requires scaling should also fall back to using
     the X Video Extension.

     This project would add support for the X Video Extension to DMX and
     would add support to VNC to take advantage of this extension
     without introducing artifacts.  This would require modifying the
     vncviewer rendering engine to cache a master copy of the desktop
     image for all operations.  This project should also add support for
     the RGB format to at least one XFree86 driver (e.g., ATI Radeon).

     The X Video Extension is one of the few popular extensions that DMX
     does not support.  We recommend implementing the X Video Extension
     even if scaling is the specific goal of that work.

  We do nnoott recommend implementation of the XCreateScaledWindow
  extension because of the complexity involved.  We do nnoott recommend
  implementation of the XPutImageScaled extension because it requires
  the same amount of work as the XCopyAreaScaled extension, but provides
  less functionality.  Further, server-side scaling with a large frame
  buffer is nnoott recommended because of the performance implications.

  The back-end scaling, especially with disconnect/reconnect support
  should be explored in the future after disconnect/reconnect is
  implemented, but not at the present time.