diff options
Diffstat (limited to 'xorg-server/hw/dmx/doc/scaled.txt')
| -rw-r--r-- | xorg-server/hw/dmx/doc/scaled.txt | 579 |
1 files changed, 579 insertions, 0 deletions
diff --git a/xorg-server/hw/dmx/doc/scaled.txt b/xorg-server/hw/dmx/doc/scaled.txt new file mode 100644 index 000000000..d30105dd2 --- /dev/null +++ b/xorg-server/hw/dmx/doc/scaled.txt @@ -0,0 +1,579 @@ + Scaled Window Support in DMX + Rickard E. Faith and Kevin E. Martin + 15 October 2003 (created 19 September 2003) + + This document investigates the possibility of adding scaled window + support to the DMX X server, thereby allowing a window or some + selected part of the logical DMX area to be displayed using a scaling + factor. For example, this might allow the contents of a window to be + magnified for easier viewing. In particular, scaling for the VNC + client is explored. _C_o_p_y_r_i_g_h_t _2_0_0_3 _b_y _R_e_d _H_a_t_, _I_n_c_._, _R_a_l_e_i_g_h_, _N_o_r_t_h + _C_a_r_o_l_i_n_a + + ______________________________________________________________________ + + Table of Contents + + + 1. Introduction + 1.1 DMX + 1.2 Problem Statement + 1.3 Task + + 2. Previous Work + 2.1 VNC + 2.1.1 Scaling under VNC + 2.2 The X Video Extension + + 3. Possible Solutions + 3.1 VNC-like Scaling + 3.1.1 Software Scaling + 3.1.2 Scaling with the X Video Extension + 3.1.2.1 Implementing the X Video Extension for DMX + 3.1.2.2 Supporting RGB formats for the X Video Extension + 3.1.3 Scaling with an XPutImageScaled Extension + 3.1.4 Scaling with an XCopyAreaScaled Extension + 3.1.5 Scaling with OpenGL + 3.2 Application-transparent Scaling for DMX + 3.2.1 Back-end Scaling Without Disconnect/Reconnect + 3.2.2 Back-end Scaling With Disconnect/Reconnect + 3.2.3 Server-side Scaling + 3.3 XCreateScaledWindow API + 3.3.1 XCreateWindow + 3.3.2 XSetWindowAttributes + 3.3.3 XGetWindowAttributes, XGetGeometry + 3.3.4 Popup and Child window positions + 3.3.5 Events + 3.3.6 Implementation + + 4. Conclusion and Recommendations + + + ______________________________________________________________________ + + 11.. IInnttrroodduuccttiioonn + + 11..11.. DDMMXX + + The DMX X server (Xdmx) is a proxy server that is designed to allow X + servers on multiple machines to be combined into a single multi-headed + X server. Combined with Xinerama, these heads can appear as a single + very high-resolution screen. Typical applications include the + creation of a video wall with 16 1280x1024 displays arranged in a + rectangle, for a total resolution of of 5120x4096. + + + + 11..22.. PPrroobblleemm SSttaatteemmeenntt + + Applications displayed on a physically large video wall that provides + high pixel-resolution may be difficult to see, especially if the + application is designed for use on a typical desktop computer with a + relatively small display located close to the human operator. The + goal of this paper is to describe and discuss solutions to this + problem. + + The original driving problem for this work is to provide scaling for + the vncviewer application when displayed using DMX (VNC scaling is + currently available only with the Windows client, and there is no plan + to extend that capability to other clients). While this specific + problem will be addressed in this paper, the general solution space + will also be explored, since this may lead to a good solution not only + for vncviewer but also for other applications. + + 11..33.. TTaasskk + + For reference, here is the original description of the task this paper + addresses: + + +o Scaled window support (for VNC) + + +o Investigate possibility of implementing a "scaled window" + extension: + + +o Add XCreateScaledWindow call that could be used in place of + XCreateWindow + + +o All primitives drawn to scaled window would be scaled by + appropriate (integral?) scaling factor + + +o Alternate approach: special case VNC support + + 22.. PPrreevviioouuss WWoorrkk + + This section reviews relevant previous work. + + 22..11.. VVNNCC + + 22..11..11.. SSccaalliinngg uunnddeerr VVNNCC + + When using the vncviewer program for Windows, it is possible to + specify a scaling factor (as numerator and denominator). When scaling + is in effect, the viewer software uses StretchBlt (instead of BitBlt) + to display the pixels for the user. When this call is made, the + viewer already has received all of the pixel information (at full + unscaled resolution). + + The scaling in VNC is primitive. It does not conserve bandwidth, it + does not treat textual information differently (i.e., by using a + suitably scaled font), and it does not provide any anti-aliasing other + than that provided by the underlying (Windows-only) system library. + + 22..22.. TThhee XX VViiddeeoo EExxtteennssiioonn + + The X Video Extension is a widely-available extension to the X11 + protocol that provides support for streaming video. Integral to this + support is the ability to arbitrarily scale the output. In version + 2.2 of the X Video specification, support for scaled still images was + provided, using both shared memory and traditional transport. The API + for this support uses calls that are quite similar to XCreateWindow, + XPutImage, and XShmPutImage. Currently, most of the drivers + implemented in XFree86 only support data in various YUV formats. + However, several modern video adaptors support RGB as well. + Note, though, that the target output for this scaling is an overlay + plane -- so X Video provides functionality that is fundamentally + different from that provided by the Windows StrechBlt call. + + 33.. PPoossssiibbllee SSoolluuttiioonnss + + This section briefly discusses possible solutions, including major + advantages and disadvantages from both the implementation and the end- + user programmer standpoint. + + 33..11.. VVNNCC--lliikkee SSccaalliinngg + + 33..11..11.. SSooffttwwaarree SSccaalliinngg + + The vncviewer application could be modified to provide software + scaling. This is not a general solution, but it does solve one of the + goals of this work. + + A prototype of this solution was implemented and a patch against + vnc-3.3.7-unixsrc is available in the dmx/external directory. Because + of limited time available for this work, all of the edge cases were + not considered and the solution works well mainly for integer scaling. + + Currently, vncviewer writes to the X display with XPutImage, + XCopyArea, and XFillRectangle. All instances of these calls have to + be aware of scaling and must round correctly. In the prototype + solution, rounding is incorrect and can cause artifacts. + + A better solution would be to cache all updates to the desktop image + in vncviewer and only send the damaged area to the X display with + XPutImage. This would allow the damaged area to be computed so that + rounding errors do not create artifacts. This method is probably + similar to what is used in the Window client. (The whole VNC suite is + being re-written in C++ and the forthcoming version 4 has not been + evaluated.) + + 33..11..22.. SSccaalliinngg wwiitthh tthhee XX VViiddeeoo EExxtteennssiioonn + + The scaling in the Windows vncviewer application makes use of a scaled + blit that is supplied by the underlying system library. Several video + cards currently provide support for a scaled blit, and some X servers + (including XFree86) expose this capability to applications via the + XvPutImage interface of the X Video Extension. The capability exposed + by XvPutImage results in the scaled image being drawn to an overlay + plane. Most video cards also provide support for a scaled blit into + the normal output planes, but this is not exposed via XvPutImage. + + The vncviewer program could be modified to use the X Video Extension + to provide scaling under X11 that is similar to the scaling currently + provided under Windows. Unfortunately, Xdmx does not currently export + the X Video Extension, so this would not provide an immediate solution + usable with DMX. + + A very early-stage proof-of-concept prototype was implemented and a + preliminary patch against vnc-3.3.7-unixsrc is available in the + dmx/external directory. This prototype was implemented to better + understand the problems that must be solved to make this solution + viable: + + +o As noted under the software scaling section above, vncviewer writes + to the X display with several different calls. These calls write + to the normal output planes and are compatible with XvPutImage, + which writes to an overlay plane. To eliminate artifacts caused by + this problem, vncviewer should be modified so that a cached copy of + the desktop is available, either as a client-side image or a + server-side off-screen pixmap, so that XvPutImage would be the only + method for writing to the X display. + + +o + + Although several modern graphics adaptors support hardware scaling + using an RGB format (e.g., ATI Radeon, nVidia, etc.), XFree86 + drivers typically only implement YUV formats. YUV generally + compress the pixel information in some way. For example, two + commonly implemented formats, YUY2 and UYVY provide intensity + information for every RGB pixel, but only provide chroma and + luminance information for pairs of horizontal pixels. Since VNC + uses pixel-resolution for communicating updates on the wire, + additional artifacts are introduced (because there may not be + enough information from the wire to update a pair of pixels). + + Further, the well-known problem with YUV encoding is even more + evident when the image is a desktop instead of a movie. For + example, consider a 1-pixel-wide vertical window border. If the + border changes in color but not intensity (e.g., because a window + manager uses color to indicate focus), there may or may not be a + change in the YUY2 image, depending on the algorithm used for RGB + to YUV conversion and on how the border pixel is ordered in the + pair of pixels used by the algorithm. + + Many of these artifacts could be eliminated if vncviewer cached a + complete RGB image of the desktop, and only did the conversion to + YUV for properly aligned areas of damage. The remaining artifacts + could be eliminated if an RGB format was used with X Video (which + may require the extension of existing XFree86 drivers to support + RGB). + + +o Most modern video cards support exactly one overlay plane that is + suitable for use with X Video. Therefore, only one application can + use X Video at any given time. This is a severe limitation in a + desktop environment. + + 33..11..22..11.. IImmpplleemmeennttiinngg tthhee XX VViiddeeoo EExxtteennssiioonn ffoorr DDMMXX + + The user-level API for X Video is fairly simple, but the underlying + support required for the full specification is large. However, since + the API provides a method to query supported capabilities, a usable + subset of X Video can be implemented that would support XvPutImage and + little else. This would require support for the following: + + +o X Video Extension API calls, including the following: + + +o XvQueryExtension + + +o XvQueryAdaptors + + +o XvQueryPortAttributes + + +o XvFreeAdaptorInfo + + +o XvListImageFormats + + +o XvGrabPort + + +o XvCreateImage + + +o XvPutImage + + +o XvShmCreateImage + + +o XvShmPutImage + + +o Support for querying back-end X Video Extension capabilities. + + +o Support for sending the image to the back-ends. Because X Video + requires sending full images, there may be a trade-off between + bandwidth limitations and additional complexity to divide the image + up such that is scales properly. + + +o Possible support for a software fall-back. For example, if all of + the back-ends do not support the X Video Extension, software + scaling can be implemented such that the image is sent to the back- + end with XPutImage. This pathway would have poor performance. + + 33..11..22..22.. SSuuppppoorrttiinngg RRGGBB ffoorrmmaattss ffoorr tthhee XX VViiddeeoo EExxtteennssiioonn + + Assuming an XFree86 driver already supports the X Video Extension, and + assuming the target hardware supports an RGB format, then adding + support for that format is relatively simple and straightforward. + + 33..11..33.. SSccaalliinngg wwiitthh aann XXPPuuttIImmaaggeeSSccaalleedd EExxtteennssiioonn + + Instead of (or in addition to) implementing the X Video Extension in + DMX, one obvious solution would be to implement a new extension that + provides access to hardware-assisted scaled blits, similar to the + StretchBlt call available under Windows. This call would scale RGB + images and would not use the overlay plane (unlike the X Video + Extension). + + This approach has many of the same advantages and disadvantages as the + XCopyAreaScaled Extension, discussed in the next section. Discussion + of XPutImageScaled is deferred in favor of XCopyAreaScaled for the + following reasons: + + +o XPutImageScaled can be emulated with XCopyAreaScaled by first using + XPutImage to copy the image to an off-screen pixmap, and then + calling XCopyAreaScaled between that off-screen pixmap and the + target drawable. + + +o Since XCopyAreaScaled would copy between two areas of on-screen or + off-screen memory, it has additional uses and can be viewed as + efficiently providing a superset of XPutImageScaled functionality. + + 33..11..44.. SSccaalliinngg wwiitthh aann XXCCooppyyAArreeaaSSccaalleedd EExxtteennssiioonn + + As noted in the previous section, because XCopyAreaScaled provides a + superset of the functionality provided by XPutImageScaled, we will + consider this extension instead. + + First, XCopyAreaScaled would provide for RGB scaling between pixmaps + (i.e., on-screen or off-screen areas of memory that reside on the + video card). Unlike the X Video Extension, which writes into an + overlay plane, XCopyAreaScaled would write into the non-overlay areas + of the screen. Key points to consider are as follows: + + +o Because different planes are involved, the two scaling operations + are usually implemented in hardware differently, so an + XCopyAreaScaled extension could be added in a manner that would + neither conflict with nor interact with the X Video extension in + any way. + + +o The XCopyAreaScaled extension provides new functionality that the X + Video Extension does not provide. Based on anecdotal feedback, we + believe that many people outside the DMX and VNC communities would + be excited about this extension. + + +o The main drawback to this extension is that it is new and needs to + be implemented at the driver level in XFree86 for each video card + to be supported. At the present time, it is more likely that the X + Video Extension will be implemented for a particular piece hardware + because the X Video extension has multimedia uses. However, over + time, we would expect the XCopyAreaScaled extension to be + implemented along with the X Video extension, especially if it + becomes popular. + + +o Another drawback is that not all modern cards provide support for a + simple scaled blit operation. However, these cards usually do + provide a 3D pipeline which could be used to provide this + functionality in a manner that is transparent to the client + application that is using the XCopyAreaScaled extension. However, + this implementation pathway would make this extension somewhat more + difficult to implement on certain cards. + + 33..11..55.. SSccaalliinngg wwiitthh OOppeennGGLL + + Another general solution to the scaling problem is to use the texture + scaling found in all 3D hardware. This ability is already exposed + through OpenGL and can be exploited by clients without X server + modification (i.e., other than the ability to support OpenGL). An + application using OpenGL would transmit the non-scaled image to the X + server as a texture, and would then display a single non-transformed + rect using that texture. This also works around the single overlay + problem with the X Video Extension as well as the need to implement + additional scaled primitive extensions. + + The downside is that most OpenGL implementations require power of 2 + texture sizes and this can be very wasteful of memory if, for example, + the application needs to scale a 1025x1025 image, which would require + a 2048x2048 texture area (even a 640x480 image would require a + 1024x512 texture). Another downside is that some OpenGL + implementations have a limited about of texture memory and cannot + handle textures that are very large. For example, they might limit + the texture size to 1024x1024. + + 33..22.. AApppplliiccaattiioonn--ttrraannssppaarreenntt SSccaalliinngg ffoorr DDMMXX + + 33..22..11.. BBaacckk--eenndd SSccaalliinngg WWiitthhoouutt DDiissccoonnnneecctt//RReeccoonnnneecctt + + VNC does scaling on the client side (in the vncviewer application). + Implementing a similar solution for DMX would require support in the + back-end X servers and, therefore, is not a general solution. + + XFree86 already implements some support for "scaling" that could be + used with DMX: if, in the XF86Config file, multiple Modes are listed + in the Display Subsection of the Screen Section, then pressing Ctrl- + Alt-Plus and Ctrl-Alt-Minus can be used to iterate through the listed + modes. The display dimensions will change to the dimensions in the + Modes line, but the logical dimensions of the X server (i.e., the + dimensions that Xdmx knows about) will not change. + + Further, the dimensions of the XFree86 display are under software + control (via the XFree86-VidModeExtension), so the Xdmx server could + change the screen dimensions on a per-display basis, thereby scaling + the information on part of that display. + + However, this scaling appears to have limited use. For example, + assume a 4 by 4 display wall consisting of 16 1280x1024 displays. If + all of the back-end servers were simultaneously configured to display + 640x480, the left hand corner of each display would be magnified, but + the composite result would be unreadable. Magnifying one display at a + time could be usable, but could have limited utility, since the result + would still be no larger than a single display. + + + 33..22..22.. BBaacckk--eenndd SSccaalliinngg WWiitthh DDiissccoonnnneecctt//RReeccoonnnneecctt + + Disconnect and reconnect features are not currently supported in DMX, + but are scheduled to be implemented in the future. These features, + combined with the XFree86-VidModeExtension Extension, would allow an + application to do the following: + + +o Disconnect a specific back-end server (via the DMX Extension), + + +o reconfigure the XFree86 back-end server resolution, and + + +o reconnect the back-end server to DMX -- at a new origin with the + new screen resolution. + + For example, consider a display wall consisting of 16 1280x1024 + displays with a total resolution of 5120x4096. All of the screens + could be disconnected, repositioned, and reconnected each at a + resolution of 640x480. The total resolution of the display wall would + be 2560x1920, allowing a view of a selected area approximately one- + fourth of the size of the DMX display. This change would be + completely application independent (except, perhaps, for a DMX-aware + window manager). When work at the increased resolution was completed, + the back-end servers could be disconnected, reconfigured, and + reconnected for the original 5120x4096 view. + + Support for this type of scaling can be implemented in a DMX-aware X11 + client assuming the DMX server support arbitrary disconnect and + reconnect semantics. Because this application cannot be written + before disconnect/reconnect is implemented, this solution will not be + discussed further in this paper. + + 33..22..33.. SSeerrvveerr--ssiiddee SSccaalliinngg + + In earlier versions of DMX, a frame buffer was maintained on the + server side, and XPutImage was used to move the information from the + server to the client (similar to some early VNC implementations). The + use of a server-side frame buffer would allow the server to do + scaling, but is not a recommended solution because of overall + performance issues and server-side memory issues (i.e., the frame + buffer would be very large for large display walls). + + Exploration of this path is not recommended. + + 33..33.. XXCCrreeaatteeSSccaalleeddWWiinnddooww AAPPII + + The implementation of X Video Extension in DMX, and the use of + XvPutImage by applications requiring scaling requires significant + changes in DMX Further, XvPutImage is, essentially a scaled blit, and + it is only useful for applications which are already using (or can be + modified to use) XPutImage. Therefore, a more general API will be + discussed as another possibility. + + X applications typically create windows with the XCreateWindow call. + A new extension could provide an XCreateScaledWindow call that could + be used in place of the XCreateWindow call and be otherwise + transparent to the application. This would allow applications, even + those that do not depend on XPutImage, to take advantage of window + scaling. In this section we describe how the call would work, what + transparency it provides, and how to solve the potential problems that + transparency creates. + + 33..33..11.. XXCCrreeaatteeWWiinnddooww + + The XCreateWindow call takes width and height as parameters. An + XCreateScaledWindow call could take all the same parameters, with the + addition of a scaling factor. + 33..33..22.. XXSSeettWWiinnddoowwAAttttrriibbuutteess + + An X11 window has several attributes that would have to be scaled: + + +o Background and border pixmaps + + +o Border width + + +o Cursor + + 33..33..33.. XXGGeettWWiinnddoowwAAttttrriibbuutteess,, XXGGeettGGeeoommeettrryy + + For transparency, calls that query the window attributes should return + unscaled information. This suggests that all unscaled pixmaps and + window attributes should be cached. + + Unfortunately, a window manager requires the scaled geometry to + properly decorate the window. The X server can probably determine + which client is acting as the window manager (e.g., because that + client will select events that are used exclusively by the window + manager). However, other Scaled Window Extension aware clients may + also need to determine the scaled geometry. Therefore, at least two + additional extension calls should be implemented: + XGetScaledWindowAttributes and XGetScaledGeometry. + + 33..33..44.. PPooppuupp aanndd CChhiilldd wwiinnddooww ppoossiittiioonnss + + Some applications may position popup and child windows based on an + unscaled notion of the main window geometry. In this case, additional + modifications to the client would be required. + + 33..33..55.. EEvveennttss + + Most events (e.g., for mouse motion) return information about the + coordinates at which the even occurred. These coordinates would have + to be modified so that unscaled values were presented to the client. + + 33..33..66.. IImmpplleemmeennttaattiioonn + + There are many implementation issues, some of which are similar to the + issues involved in implementing the X Video Extension for DMX. The + window contents must be scaled, either by performing all operations to + a frame buffer and then writing the image to the display (perhaps + using hardware scaling support), or by modifying all of the various + drawing operations to perform scaling. Because of the complexity + involved, the frame buffer option is recommended. + + 44.. CCoonncclluussiioonn aanndd RReeccoommmmeennddaattiioonnss + + We recommend a three phase implementation strategy, based on how an + application could be written to take advantage of scaling: + + 1. + + The XCopyAreaScaled extension should be implemented, since this is + the ideal solution for applications like VNC, and since making use + of this extension will require minimal changes to applications that + already use XPutImage or XCopyArea. + + The initial implementation work would include the design of the X + protocol extension, writing this up in the usual format for + extension documentation, implementation of the protocol transport + pieces in XFree86, implementation of a software fall-back in + XFree86 and DMX, one example hardware implementation for XFree86, + and implementation of support for this extension in DMX. + + We suggest implementing the extension first on the ATI Radeon + cards. However, since these cards do not provide a 2D scaled blit + primitive, the implementation would have to make use of the 3D + texture engine to emulate a scaled blit. This is recommended, + since other modern graphics cards also do not provide a simple 2D + scaled blit operation and an example of the more difficult + implementation pathway would be helpful to others. + + 2. + + Until XCopyAreaScaled is widely supported, applications that + require scaling will have to fall back to another scaling method. + We suggest OpenGL as the first fall-back method because it is + widely available and supported by DMX. + + A project centered around OpenGL-based scaling would implement this + scaling in VNC as an example. This work would include re-writing + the vncviewer rendering engine to cache a master copy of the + desktop image for all operations. + + 3. + + Since OpenGL is not implemented everywhere, and may not provide + hardware-assisted performance in every implementation, an + application that requires scaling should also fall back to using + the X Video Extension. + + This project would add support for the X Video Extension to DMX and + would add support to VNC to take advantage of this extension + without introducing artifacts. This would require modifying the + vncviewer rendering engine to cache a master copy of the desktop + image for all operations. This project should also add support for + the RGB format to at least one XFree86 driver (e.g., ATI Radeon). + + The X Video Extension is one of the few popular extensions that DMX + does not support. We recommend implementing the X Video Extension + even if scaling is the specific goal of that work. + + We do nnoott recommend implementation of the XCreateScaledWindow + extension because of the complexity involved. We do nnoott recommend + implementation of the XPutImageScaled extension because it requires + the same amount of work as the XCopyAreaScaled extension, but provides + less functionality. Further, server-side scaling with a large frame + buffer is nnoott recommended because of the performance implications. + + The back-end scaling, especially with disconnect/reconnect support + should be explored in the future after disconnect/reconnect is + implemented, but not at the present time. + + + |