impl

Multi-Screen Mouse and Optical Mouse

R. Negrin, A. Edwards, J. Murray

Project Implementation

The implementation of the project can be divided into several sections. These include the configuration dialog, the server program, the communication layer, the client program and dialog, and the optical mouse. The implementation information for each of these parts is included in this part of the report.

Internet Mouse Configuration Dialog

The configuration dialog is an MFC based application written in C++. Figure 1 is a screen collage of the dialog. Table 1 describes the features of the collage, how to use each feature, and talks a little bit about the implementation of each feature.

Figure 1. Internet Mouse Configuration Dialog

Table 1. Configuration Dialog Usage and Implementation.
Feature	Usage	Implementation
Hide Button	Press this button to hide the configuration dialog.	The configuration dialog is hidden with the Win32 API ShowWindow. To get the dialog back, click on the mouse icon on the taskbar, or re-launch the application. The dialog is a single instance application. Subsequent instances will show the original instance and the exit. Single instance behavior is implemented via a system semaphore on Win32.
Exit Button	Press this button to exit the application.	This button exits the application. Once the application is exited, the Internet Mouse will not function. Configuration information is stored in the registry when the application exits. Restarting the application will resume in the previous configuration.
Screen Arrangement Area	Click and drag to move screens around.	The screen arrangement area, which is outlined in black, allows the user to input the physical arrangement of the screens. Positioning one screen to the left of another will allow the user to move off the left edge of one screen onto the right edge of the other screen. For example, in Figure 1, moving the mouse off the right edge of "WAFFLE" will land the user on "Pancake." Changes to the configuration are reflected immediately in the behavior. In order to reduce flickering when dragging, we use a background buffer scheme. A pink screen in this area indicates that the cursor is currently on that screen. A gray screen indicates a client that is not currently available. A white screen indicates a client or server that does not have a cursor on it at that instant. Note that right clicking on a screen will bring up a context menu, and that this area of the dialog will resize itself to make room for more screens.
Add Screen Button	Press this button to add a screen to the screen arrangement area.	When the user presses this button, they are presented with the "Screen Properties" dialog for the new screen. At that point, the user should enter the friendly name of the computer. The width of the screen is also required for proper operation. For more information, see the Screen Properties entry in this table.
Delete Context Menu Item	Select this item from the context menu to delete a client screen.	When the user brings up a context menu by right clicking on a client screen, and then selects delete, the screen they clicked on is deleted from the arrangement, and the client is disconnected. To get the screen back, add it again.
Properties Context Menu Item	Select this item to bring up the "Screen Properties Dialog" for a screen.	When the user brings up a context menu by right clicking on a client screen, and then selects properties, they reach the properties dialog for that screen. For more information, see the Screen Properties Dialog entry in this table.
Screen Properties Dialog	This dialog allows a user to configure a screen.	The user can change the computer that they are arranging with this dialog, or correct the size of a screen. Pressing "Apply" changes the information in the screen arrangement area, and will result in an attempt to connect to a client computer if there is not already a connection. Pressing "Close" results in an implicit apply, and closes the dialog.
Taskbar Icon	This icon will make the configuration dialog visible.	The user can click on this Icon to bring the configuration dialog to the top of their desktop or to cause the dialog to be shown when it is hidden. The icon is added to the taskbar through the Win32 API Shell_NotifyIcon. When the user clicks on the icon, we receive a message from windows indicating that the user clicked on the dialog.

Internet Mouse Server

The Internet mouse server is in charge of the accepting mouse messages and sending them to the right machine. It must also watch the client machines to make sure they are still working. The distinct features and their implementations are described in Table 2.

Table 2. Internet Mouse Server Details
Feature	Description and Implementation
Capturing Mouse Messages	The server captures the mouse messages through a system journaling hook. The hook is installed by calling SetWindowsHookEx, and passing in a pointer to a function which should be called whenever a mouse message comes to the system. When this function is called, we pass the mouse message off to the network communication layer if the cursor is currently on the server. This hook does not provide a means to block the messages from going to other application running on the server. For a description of how we implemented that, see the section "Stopping Mouse Messages" in this table.
Showing and Hiding the Cursor	To show and hide the cursor on a system wide level, we had to hide it for a window and put that window over the entire screen, which has no visible cursor. In order to make things look right, we create the window over the entire screen with the WX_EX_TRANPARENT style so that it has no visible effect on the appearance of the other applications. When the cursor needs to be visible, we hide this window. When the cursor needs to be hidden, we show this window.
Stopping Mouse Messages	We also use the transparent window to prevent mouse message from going to other applications running on the server machine when the cursor is on another machine. The transparent window just ignores these mouse messages.

Network Communication

We needed a simple network interface for sending the mouse messages across the network and between computers. The obvious choice was sockets based on TCP/IP. The protocol consists of a number of messages, as described in Table 3.

Table 3. Internet Mouse Messages
Message	Description
MOUSE_MOVE	Mouse activity. Followed by two 16-bit integers containing the offsets for the x and the y direction.
MOUSE_LBUTTON_DOWN	Mouse activity.
MOUSE_RBUTTON_DOWN	Mouse activity.
MOUSE_LBUTTON_UP	Mouse activity.
MOUSE_RBUTTON_UP	Mouse activity.
ENTER_SCREEN	The server sends this message to the client as the mouse moves onto the client. The next byte indicates which edge the mouse is coming from (0 is right, 1 is left, 2 is top, and 3 is bottom). The next 32 bits give the offset along the edge where the mouse is entering. The client should show the mouse cursor.
EXIT_SCREEN	The server sends this message to the client in response to an EDGE_SCREEN message when there is another screen to go onto. The client should hide the mouse cursor.
ARE_YOU_ALIVE	The server sends this message to the client to see if the client is alive. The client should respond with an IM_ALIVE message.
SHUTDOWN	The server sends this message to the client when the screen is removed from the list of available screens.
ACCEPT_CONN	The server sends this message to the client when the client tries to connect to the server and the client is listed as a valid client.
REJECT_CONN	The server sends this message to the client when the client tries to connect to the server and the client is not listed as a valid client.
IM_ALIVE	This is the client’s response to the ARE_YOU_ALIVE message sent by the server. If the client does not respond to this message for an extended period of time, the server will not allow the user to move the mouse onto the screen, and it will remove the mouse from the client if the mouse was already there.
EDGE_SCREEN	This is the message the client must send to the server when it detects that the user has moved the mouse to the edge of the screen. The next byte indicates which edge the mouse is on (see ENTER_SCREEN), and then the next 4 bytes is the position along the edge. The server may respond with an EXIT_SCREEN if there is another screen to move to.

We kept the client as simple as possible. The only information the client needs to know is whether it has hit an edge. If it has hit and edge, it has to notify the server. The server does not know if a client has hit an edge because the server does not send absolute positions to the clients, it sends deltas. It would have been nicer to send absolute (virtual coordinates) to the client from the server. This would have made the client even easier, but there are two problems with virtual coordinates. First, if you had a server with a small screen and a client with a big screen, the client's mouse movement would have been faster because the virtual coordinates would have been a percent of the screen the mouse moved. Since the client has a bigger screen the percentage will be a bigger number, and the mouse on the client would have been too fast. Also, the client would have had to notify the server when another application, like the "snap-to-button" programs some people use on laptops, moved the mouse cursor. Therefor, the server sends virtual coordinates, and the client notifies the server when it has hit an edge. At that point, the server decides whether control should switch to another screen or stay on the current one.

Keeping the client simple was a priority so that clients could be written for older operating systems that are difficult to do real-time low level programming. For example, Windows 3.1 or older versions of the Macintosh.

Win32 Internet Mouse Client and Configuration Dialog

The Win32 Internet Mouse Client is a dialog application written using MFC and VC++. The dialog, shown in Figure 2, allows the user to set the server that will control the mouse on the client. The "Apply" button causes the client to attempt to connect to the computer specified in the server edit box. The hide button hides the dialog, which can be shown by clicking on the taskbar icon, or re-launching the client because the client is a single instance application like the Win32 server. The exit button exits the client application.

Figure 2. Internet Mouse Client Configuration Dialog.

The Win32 client has several duties, which are common to all clients. These duties are outlined in table 4.

Table 4. Client duties.
Duty	Description
Connecting to the server.	When the client starts up, it needs to attempt to connect to the server on port 1069. If a connecting is made, the client should start receiving messages and doing the appropriate action for the messages. These actions are all described in this table.
Responding to ARE_YOU_ALIVE messages.	The server sends an ARE_YOU_ALIVE message to clients when it wants to know if they are alive. The client should respond by sending an IM_ALIVE message to the server.
Responding to the ENTER_SCREEN message.	When the server receives the ENTER_SCREEN message, it should make the cursor visible at the appropriate position (which is sent after this message, see Table 3 for details.
Responding to mouse action messages.	When the server receives a mouse action message, it should simulate that mouse action on its screen. The client is also in charge of checking to see if the mouse has reached the edge of the screen. If the mouse is at the edge of the screen, it should send a EDGE_SCREEN message to the server, as described in Table 3.
Responding to the EXIT_SCREEN message.	When the mouse is at the edge of the screen, and the client has sent an EDGE_SCREEN message to the server, the server may respond with an EXIT_SCREEN message. At this point, the client needs to hide the cursor.

The implementation of the Win32 client has several details that are unique to the Win32 platform. These details are included in Table 5.

Table 5. Win32 Client Implementation
Feature	Implementation
Hiding the mouse.	On Win32, hiding the mouse is accomplished with a transparent window that covers the whole screen. This window is hidden when the mouse should be visible.
Network communication.	Network communication is implemented with a background thread. The thread sits in a loop waiting for messages from the server. When it receives a message, it performs the appropriate action.
Simulating mouse movements.	On Win32, mouse movements are simulated with calls to the mouse_event API.

Mac Internet Mouse Client

The first thing we had to do on the Macintosh was find a TCP/IP implementation we could easily use. We were most familiar with Berkeley sockets, so we tried using SK_sockets from the Decision Group at Harvard University. However, we had some problems getting the asynchronous routines working and that package was poorly documented. Next, we switched over to MacTCP. Although the documentation was good, there were not many examples. We were able to leverage off some code we found on the Internet, and with a little experimentation we figured out how it worked.

The next thing to do was figure out how to manipulate the mouse on global level. The documentation was scarce and inaccurate. For the older Macintoshes, those without ADB (Apple Desktop Bus), you have to write to global memory variables to move the mouse. We successfully implemented mouse movement with these global variables, although we never got button clicking working properly. We know it is possible, but we never did it.

For the new Macintoshes that support ADB, there are cursor device functions that support moving the mouse and clicking the button. We used these functions in our actual implementation. The code is not portable to the older Macintoshes because of these functions. However, the code to move the mouse and simulate button clicks is isolated in its own module and could be replaced by code that uses the low-level global variables to move the mouse and simulate the button clicks.

Once we had a TCP engine that worked and understood how to manipulate the mouse we wrote a user level application that implemented a client. This client's performance was terrible. The mouse movement was choppy and unusable. While a large program was loading the mouse was completely locked up. We would also lock up the machine if we used blocking sockets, so we had to use non-blocking sockets. To solve the performance problem we used non-blocking sockets with an asynchronous receive. What this means is that when we are ready to receive data we call receive and it returns immediately and we just go on with our application. When data arrives, a user defined function is called, which was passed in when we called Receive. This function does the processing of the data and then calls Receive again so that it will be called when more data arrives.

The advantage of asynchronous sockets is that data gets processed immediately after it arrives and the operating system is free for use in-between receives. (Remember we only receive stuff as fast as a mouse can send it which is pretty slow relative to most networks). This sounds ideal, however the call-back function is called during an interrupt. During an interrupt there are many things you cannot do, including allocating, delete, or moving memory. In addition, you can not make blocking calls to Send or Receive, nor can you make more than one asynchronous receive call or asynchronous send call. These limitations complicated our Mac client. The way the protocol works is that we receive a control message that specifies what kind of information would follow. For example a mouse move message would indicate that two sixteen bit numbers are going to follow, representing the horizontal delta and the vertical delta. On the other hand, a button click message has no data following it. After the data or a control message with no data, the next data received is another control message.

This requirement mapped nicely to a state machine. We start in the waiting-for-a-control message state. When a message arrives, we go to a state that depends on the control message. (For example if we get a mouse move message, we would go to the first mouse move state.) From there we proceed through a number of states until we collect all the data for that message. We then return to the waiting-for-control-message state. If the control message that comes in has no data (like like button up event), we stay in the waiting-for-control-message state. The Mac is currently implemented with a state machine similar to this, except that our Receive only collects one byte at a time. We did this to simplify the code, so that we do not have to call Receive with different size buffers depending on the state. Receiving only a byte at a time reduces complexity, but it increases the number of states because we now have to have a state for every byte of data in the control message. This is not the best way to implement a client, because it does not scale well to control messages that might be implemented in the future. An example would be sending the contents of the clipboard, where we could be sending many bytes which would have a significant amount of overhead.

The specifics of the Mac client implementation are summarized in Table 6.

Table 6. Mac Client Implementation
Feature	Implementation
Hiding the mouse.	This feature is not implemented on the mac. The mouse cursor is always visible. When the mouse is on another screen, the cursor sits on the edge of the mac and does not move.
Network communication.	Network communication is implemented with asynchronous receiving. We are called when a byte arrives from the network, and perform the necessary action based on our current state and the contents of the byte that arrived. Receiving the data in an interrupt was necessary because the Mac OS does not preemptively multitask.
Simulating mouse movements.	There is an API on Macs with ADB (Apple Desktop Bus) to simulate mouse movements and we call that. Because of this API, the Mac client is not portable to earlier versions of the Mac.

The Mac client GUI has a dialog box with a connect button, a close button, and an edit text box. The user enters an IP address in the text box, and then presses connect. If the connect is successful the IP address is stored in a file. The next time the program is started that address is loaded into the dialog box, so that the user does not have to type in the IP address every time. We assume the server will not need to change very often. When the user presses the close button the connection is closed and the dialog goes away. If the user wants to start up the Internet Mouse Client again, he must run the program again. In addition, if the server exits, the dialog closes down. It does not make sense for the client to hang around if the server is not up.

The Mac GUI is so simple because we ran out of time. It took so long to get the network engine running, we did not have enough time to make a nice GUI. Although programming a GUI on the Macintosh is not difficult it does have a learning curve which we did not have time for. The current GUI is functional, but could be improved. For example, instead of closing when a connection is broken, it could hang around and try to reconnect. Another improvement would be to store the IP address somewhere in the system like the preferences folder, so that the user would not have to keep an initialization file in the same directory as the Mac client executable.

Optical Mouse

Data Format

The Xerox Optical Mouse uses a scheme called quadrature encoding to send mouse movements to the host computer. Each dimension is represented by two signals. The X direction is represented with XA and XB, the Y direction with YA, YB. Figure 3 shows how movement is represented in the X and Y direction.

Figure 3. Quadrature scheme (taken from Xerox Optical Mouse manual).

We can think of the quadrature scheme as a bit-stream, as shown in Figure 4. Here we just look at the x direction:

XA - 1111000011110000111100001111
XB - 0011110000111100001111000011
            ^^^^^^^^
            full cycle of pulses

Figure 4. Quadrature as a bit-stream.

This bit stream can be decoded into four discrete bit pairs for the two sensors, as shown in Figure 5.

       Forward direction     Backwards direction
            0 1 2 3                3 2 1 0
       XA - 1 1 0 0                0 0 1 1
       XB - 0 1 1 0                0 1 1 0

Figure 5. Bit stream decoded into discrete bit pairs.

These bit pairs can easily be decoded inside of a simple micro-controller.

Optical Mouse Pin-out

We next had to determine the pin-out used by the Xerox Optical mouse. We initially found ground and VCC by backward engineering the circuit board contained in the mouse. We could then power up the mouse to determine the other signals by simply looking at a logic analyzer output. The pin out of the mouse is shown in Figure 6.

+----------------------------+
 \  1  2  3  4  5  6  7  8  /
  \   9 10 11 12 13 14 15  /
   +----------------------+

1 - XA
2 - XB
3 - YA
4 - YB
6 - VCC
11- ground
12- button1
13- button2
14- button3

Figure 6. Pin out of the Xerox Optical Mouse.

PC Interface

There are three major ways that mice interface with PC's.

PS/2 Protocol
Serial Protocol
Bus Mouse Protocol

An excellent discussion of each of these protocols can be found at http://www.hut.fi/~then/mytexts/mouse.html. The most important thing to note is that the ps/2 and serial protocol are based on serially transmitted packets and that the bus mouse protocol is based on a quadrature encoding scheme.

The obvious choice was to use the bus mouse protocol. We first located an old bus mouse card and then wired the signals from the optical mouse to the bus mouse inputs. The bus mouse pin out is shown in Figure 7.

  -------
 / 1 2 3 \
| 4 5 6 7 |
 \  8 9  /
  -------

1. button2
2. button3
3. ground
4. XB
5. YA
6. YB
7. button1
8. VCC
9. XA

Figure 7. Bus mouse pin out.

Last Update: 03/16/97