Ubuntu Wiki

Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

• Launchpad entry: https://features.launchpad.net/distros/ubuntu/+spec/mouse-tweaks

• Packages affected:

Summary

The standard functions of the pointer (usually the mouse) has not much changed over the years. It is used to point, to select, to click&drag and to open the contextual menu. By adding tweaks and new functions to the pointer, we can improve the experience of two categories of people:

• People that have difficulties using a normal mouse will have more ways to operate the pointer.
• Powerusers have a pointer that offers more functions.

MouseTweak #1:

left click&hold to open the contextual menu (i.e.: the menu that the user gets with the right click)

MouseTweak #2:

dwell clicking: i.e.: make a left click, a left double click, a right click and a left click&drag without pressing any mousebutton.

MouseTweak #3:

tremor suppresing

MouseTweak #4:

mouse gestures: i.e.: triggering of specific actions by doing gestures with the mouse

Rationale

MouseTweak #1:

There are users that can only operate one mousebutton. These users have to find an alternative way to make the contextual menu appear. If they could make the contextual menu appear with a click&hold of their unique mousebutton (usually being the left mousebutton), they would have a direct and efficient way to access the contextual menu.

MouseTweak #2:

Users that can move the pointer but that cannot press any button are not able to do anything on a computer. They need a way to click without pressing a button. Dwell clicking is one solution to their problem because the dwell clicking software does the clicks without having its user press any button.

MouseTweak #3:

There are users that do erratic movements when mouving the mouse and/or that are not able to keep the mouse completely motionless. These user would welcome an option in the mouse preferences that would smooth the movements they do with the mouse.

MouseTweak #4:

Some users that like to control their computer with the mouse would appreciate if it was possible to do more with it. The mouse gestures software allows them to define specific actions to specific gestures that they do with their mouse. Moreover, having a more powerful mouse will also come handy for disabled users, because it gives them more choices to operate their computer.

Use cases

MouseTweak #1:

• James is a disabled user that is only able to use the left mouse button. Every time he wants to open the contextual menu (which is usually opened with a right click), he has to do it in an indirect way: for example by selecting switch button in onboard. If the contextual menu appeared with a left click&hold, he would have an efficient and direct way to do it.

• Cindy uses a tablet pc, that she controls mainly with a pen. The left click&hold would enable her to open the contextual menu with the pen.

• It could also be useful on information terminals operated by touchscreens.

• People running Ubuntu on an Apple PowerMac (PPC) traditionally have a mouse with only one button (I don't know if it has changed with the latest models). They will probably welcome a direct method to open the contextual menu.

MouseTweak #2:

• Michael is a disabled user that uses an eyetracking device to move the pointer. However the device does not provide a way to do the clicks. Combined with dwell clicking, he gets the same functionality of a normal 2 button mouse.
• Mary uses a headpointer to move the mouse on the screen. Her headpointer comes with dwell clicking software that only runs on Windows. She needs an equivalent software that would allow her to use her headpointer also on Ubuntu.

MouseTweak #3:

• Albert is a silver surfer that is over 70 years old. As his hand has gotten tremblier with the age, he would appreciate if the system could ignore part of the erratic movements that he does with the mouse.
• Victor uses a headpointer to move the pointer on the screen. However he has difficulties to keep the pointer motionless for several seconds.

MouseTweak #4:

• Anna is a poweruser that recently switched from Windows to Ubuntu. She was accustomed to use the mouse gestures software in Windows and would be happy to find an equivalent on Ubuntu.
• Jim is a disabled user that is able to move the mouse but unable to click. By using the mouse gestures software defined below in timed mode, he will not only be able to do the various clicks, but he also has many commands available by merely doing mouse gestures.

Scope

MouseTweak #1:

The left click&hold to open the contextual menu should work systemwide. (for all contextual menus, even at login if appropriate)

MouseTweak #2:

The dwelling functionality has to work systemwide.

MouseTweak #3:

The tremor smoothing functionality should be available systemwide.

MouseTweak #4:

The mouse gestures functionality should be available everywhere, even at login.

Design

MouseTweak #1:

• This mousetweak opens the contextual menu of the item that lies under the pointer, when the user keeps the first mousebutton pressed for a determined amount of time without moving the pointer.

• In order to avoid conflicts with the click&drag, the system could proceed like this: if after the mousedown, the mouse does not move for n milliseconds (for example specified with a slider), open the contextual menu; if the mouse moves before the specified time has elapsed, go for a drag&drop.

• Of course, the current n has to be greater than the current double-click time; for simplicity reason, one could also generally make: minimum(n) > maximum (double-click time), as I did in my example tab below.

• The time n of opening the contextual menu 1.5 seconds after the mousedown is probably a good default value.
• The contextual menu stays opened as long as the users keeps the left mousebutton pressed.
• When the user releases the left mousebutton, the command of the contextual menu under the pointer is executed and the contextual menu is closed. If the pointer is not over the contextual menu when the button is released, the contextual menu is closed without anything else happening.

• The left click&hold for the contextual menu should be optional. A check box in the mouse control panel and/or in the accessibility control panel should allow the user to enable and disable the function. Its default shipping state should be disabled.

• There is also a checkbox to automatically enable this function at login. (We could drop the 'Automatically enable...' checkbox and make the next session use the same settings of the previous session.) If available, this checkbox should be disabled by default.

• Below you can see a sample preference window built with 'Glade Interface Builder' that would offer a left click&hold; it is simply the 'Buttons tab' of the Mouse Preferences shipped with Ubuntu Edgy to which I have added (among others) a frame for the left click&hold.

  • inline:MousePrefButtonsTabV05.png

MouseTweak #2:

• Dwell clicking would belong to the accessibility features of the mouse.
• See below for a sample configuration dialogue for dwell clicking:

  inline:MousePrefDwellingTabV05.png

• The checkbox 'Enable Dwelling' is used to enable the dwell clicking for the account of the current user. Its default state should be disabled.
• The checkbox 'Automatically enable after login' will automatically enable dwell clicking for that user after he logs in. Its default state should be disabled.
• The user can choose between 2 ways of doing the dwell clicking (radio buttons in the configuration dialogue):

  1. Dwell clicking with the help of a window to choose between left click, right click, double click and drag&drop. (Dwelling with ClickType Window). This is the default dwelling type when active.

     inline:DwellingClickTypeSelectionWindow.png

     • When dwelling is enabled, and this way of dwelling is selected, the 'Select Click Type' window appears on screen.
     • The system clicks on the pointers location, if the pointer stays without moving on that location for the time set in 'Time to wait before click' (let us call this time the dwell delay). A dwell delay of about 1.5 seconds is probably a good default value.
     • The type of click which is performed corresponds to the click type that is set in the 'Select Click Type' Window.
     • To change click type, the user has to move the pointer over the button that corresponds to the click type that he wants, and wait for the dwell delay to elapse without moving. The system then activates the button, regardless of what click type is currently set.
     • If the checkbox 'Automatically switch back to single click mode' is activated, the dwell clicking function automatically reenables the left click after another click type has been performed. If this checkbox is not activated, the current click type does only change when the user selects another click type in the 'Select Click Type' window. The checkbox is activated by default.
     • The 'Drag And Drop' click type works like this (This is also used for example to select text):
       • the user moves the pointer over the item to drag
       • after the dwell delay has elapsed, the item is grabbed (mouse button down)
       • the user sees the grab because the item gets selected or better: we could make the pointer change its shape
       • after the grabbing, the system watches for the pointer to be move for another dwell delay. Two situations are possible:

         • the pointer does not move before the second dwell delay elapses: the drag&drop did merely correspond to a long left click. The drag&drop terminates when the second dwell delay has elapsed. (Textmark A)

         • the pointer moves before the second 2 dwell delay elapses: depending on the context: an item is dragged, text is selected,... The drag&drop terminates only after the pointer has been motionless for another dwell delay. (In other words: if during the dragging, the pointer stops shortly and restarts moving before the dwell delay has elapsed, the dragging continues.)

     • Detailed description of how to use the contextual menu when dwell clicking:

       • The user chooses the right click type in the ClickType Window

       • Then he moves the pointer over the item whose contextual menu he wants to open
       • After the pointer has been motionless over the item for the dwell delay, the contextual menu opens.
       • If I am correct, the contextual menu in gnome remains open until the next click; we don't need to change this when dwell clicking. We have only to be aware that after the click to open the contextual menu, we have to move the pointer to have another click performed after the pointer stops again for the dwell delay.
       • Depending of whether the checkbox 'Automatically switch back to single left click mode' is enabled or not, the click after opening the contextual menu is a left click if the checkbox is enabled or a right click if the checkbox is disabled. As Ubuntu allows both clicks to choose a contextual menu item (or am I wrong?), it works regardless of the setting of that checkbox. It is also possible that the click after the opening of the contextual menu is performed outside the contextual menu. In this case, I don't see any reason for the system not to behave as if it was a left/right (depending on the checkbox) click of a normal mouse.
  2. Dwell clicking where the type of click is determined by time and a gesture:
     • When this type of dwelling is used, the pointer cycles between 2 states: the pointer state and the dwell state; the user can determine the state of the pointer from its shape: in pointer state we see the usual arrow and in dwelling we see the dwell shape.
     • The cycling only occurs when the pointer is motionless.
     • The dwell state appears after the pointer has been motionless for the time determined by the "delay before dweelling starts" slider. A good default value might be 1.5 seconds.
     • The dwell state remains for the time determined by the "duration of mouse/dwell state" slider. After that time has elapsed, the pointer changes back to pointer state for the time determined by the "duration of mouse/dwell state" slider. After another delay of the time determined by the "duration of mouse/dwell state" slider has elapsed, the pointer switches back to dwell state... And so on as long as the pointer remains motionless. A good default value might be 1.5 seconds. For simplicity reasons, we could drop the second slider and set "delay before dweelling starts" equal to "duration of mouse/dwell state".
     • If the pointer begins to move when it is in its pointer shape, it is a normal mouse movement.
     • If the pointer begins to move when it is in the dwell shape, the system checks in what direction the movement began:
       • If the movement started in the left direction, perform a left click on the coordinates of the screen where the pointer was motionless
       • If the movement started downwards, perform a left double click on the coordinates of the screen where the pointer was motionless
       • If the movement starts to the right, perform a right click on the coordinates of the screen where the pointer was motionless; there might be an item with a contextual menu at these coordinates (as the usual behaviour of the contextual menu is to remain open until the next click, it is also possible to work with contextual menu when dwell clicking with gestures)

       • If the movement started upwards, it is a drag&drop (which corresponds to a mouse button down):

         • In order to avoid conflict with pointer shapes depending on the context (for example caret in text selection), it might be wise for the pointer to drop its dwell shape as soon as the drag&drop has started. (Or add a checkbox to the dwelling configuration dialogue and let the user choose whether he wants the pointer to keep its dwell shape until the end of the drag&drop.)

         • Depending on the context, grab the item under the coordinates where the pointer was motionless and drag it around, or begin selecting text, or...

         • To terminate the drag&drop (which in fact is the emulation of the mouse button up), the pointer has to stay motionless for a determined delay (which could again correspond to "delay before dweelling starts". (put the dragged item where the pointer stopped moving, or end the selection where the pointer stopped moving, or...)

       • After performing the left click, left double click, right click and drag&drop, the pointer returns to its pointer state and the cycling begins again with the "delay before dwelling starts" time.

MouseTweak #3:

inline:MousePrefTremorTabV05.png

MouseTweak #4:

inline:MouseGesturesConfigurationV04.png

Above you can see a sample GUI for the mouse gestures functionality of the pointer. There are 3 ways for doing mouse gestures:

1. by clicking a mouse button and optionally holding a modifier

   • The user can choose between a left button single click&hold, a left button double click&hold, a right button click&hold and a middle button click&hold.

   • The "Click normally if there is no movement in:" setting is especially useful to distinguish a click&drag from the mouse gestures.

   • If the gesture takes more time than the timeout allows, the gesture is ignored.
2. by indicating the gesturing with a key
   • Of course, in this mode, the user should use a key that he does not need in normal usage. A Function key or a modifier could be good candidates
3. by starting the gesture in determined time intervals.
   • When gesturing in this mode, the pointer has 2 different states:
     • Pointerstate: the pointer behaves as a normal mouse when it is moved and clicked.
     • Gesturestate: if the pointer is moved while in gesturestate, the movement is interpreted as a gesture.
   • The state of the pointer at the start of the movement determines whether it is a mouse gesture or not.
   • When the pointer is motionless, it toggles continually between pointerstate and gesturestate.
   • The pointer keeps the state it had at the start of the movement as long as it is moved.
   • The length of the pointerstate+gesturestate can be defined by the cycle parameter.
   • There is also a parameter available for the user to define how much time of the cycle the pointer has to be in pointerstate; the rest of the time of the cycle the pointer is in gesturestate. The cycle begins in pointerstate.
   • At the end of a movement (regardless of it being a gesture or not), a new cycle begins. There is one exception: if the action of a gesture is "toggle visibility of the pointer", the pointer remains in gesturestate as long as the pointer is invisible.
   • The user is able to distinguish visually whether the pointer is in pointerstate or gesturestate; for example by changing its shape or colour.
   • The "Change the colour of the path when the gesture is not recognized" checkbox is greyed when the "Draw the path of gesture" checkbox is disabled.

When clicking on the "Define Gestures..." button, the user gets to the dialogue where he can define the mouse gestures for all and for specific applications:

inline:MouseGesturesDefinitionDialogueV04.png

• On the left in the Mouse Gestures Definition Dialogue the user can add and remove applications for which he wants to define gestures.
  • The Global and Dialogue entries of the list cannot be removed.
  • The gestures defined in global work in all the applications that are in the list.
  • If the checkbotton "Enable global gestures in unlisted applications" is enabled, the gestures defined in global also work in the unlisted applications.
  • To disable gestures in an application, the user has to add it to the list and enable the "Disable all gestures in this application" checkbox.
  • If the user defines a gesture in an application and the same gesture is also defined in global, the gesture definition in the application overrides that in global; in other words, the action defined in the application is executed and the action defined in global is ignored.
  • When the "Disable global gestures for this application." checkbox is enabled, only the gestures defined for that application trigger an action.
  • The "Disable global gestures for this application." and the "Disable all gestures in this application" are not visible when the user selects Global or Dialogue in the list of applications.
  • In the Dialogue entry in the list of applications, the user can define gestures that are available when the front window is a dialogue. These gestures override the gestures defined in Global and in the application.
• On the right in the Mouse Gestures Definition Dialogue the user can see the list of gestures that he has defined for the application he has selected in the list of applications.
• When a gesture is performed by the user, the system first checks whether that gesture is defined in the list of gestures for the active application. If a match is found, the corresponding action is executed. If there is no match, the search goes on in the list of gestures defined in global and the action defined in global is executed if a match is found here. Is there is no match in the active application nor in global, nothing is done except for the changing of the colour of the path if the corresponding checkboxes are enabled.
• He can add, edit and remove gestures for the selected application by clicking on the appropriate button in the Gestures Definition Dialogue.
• By clicking on the button to add or edit gestures, the following dialogue appears:

inline:MouseGestureEditingDialogueV04.png

• In the first field, the user can give a name to the gesture.
• With the left, down, right and up buttons, the user defines the movements of the gesture.
• With the popup, the user defines the action that is executed by that gesture. The popup provides the following choices:

inline:MouseGestureEditingPopupMenuV04.png

• Some actions require a parameter, others not. Here is a more detailed description:
  • gda: Do Nothing: no parameter
  • gda: Ignore Next Gesture: no parameter
  • gda: System Volume Up: no parameter
  • gda: System Volume Down: no parameter
  • gda: Toggle Mute: no parameter
  • gda: Close Window: two radiobuttons: first: close the window where gesture occurred; second: close the front window
  • gda: Minimise Window: two radiobuttons: first: minimise the window where gesture occurred; second: minimise the front window
  • gda: Zoom Window: two radiobuttons: first: zoom the window where gesture occurred; second: zoom the front window
  • ga: Next Window: no parameter
  • ga: Previous Window: no parameter
  • ga: Quit Application: two radiobuttons: first: quit the application where the gesture occurred; second: quit the front application
  • ga: Next Application: no parameter
  • ga: Previous Application: no parameter

  • ga: Execute MenuItem: text entry field (single line) to enter name of menu item to execute

  • gda: Execute Keystroke: text entry field (single line) to enter keystroke to execute
  • gda: Type Text: text entry field (multiple lines) to enter text to type
  • ga: Open Selectiom: no parameter
  • gda: Open App/Doc: text entry field to enter name of app/doc and a button that opens the choose file dialogue to select the app/doc
  • gda: Open URL: field to enter url
  • d: Press OK: no parameter
  • d: Press Cancel: no parameter
  • d: Press Apply: no parameter
  • gda: Left Click: no parameter
  • gda: Left Double Click: no parameter
  • gda: Right Click: no parameter
  • gda: Drag and Drop: no parameter
  • gda: Toggle Pointer Visibility: no parameter
  • Remark:
    • the popup menu in the Gesture Editing Dialogue has different entries for the global/application gestures and the dialogue gestures.
    • gda means: g=entry available in the global popup menu, d=entry available in the dialogue popup menu, a=entry available in the application popup menu
    • to have a detailed description about how the click actions work, please refer to the dwelling with gestures specification of mousetweak#2.

Implementation

I don't know whether it can be useful, but in the following zip package you can find the files that I created with glade to present the sample gui:

attachment:mousetweaks.gladeversion2.12.1.zip

MouseTweak #1:

Code

MouseTweak #2:

Code

MouseTweak #3:

Code

MouseTweak #4:

Code

Unresolved issues

MouseTweak #1:

• Is the contextual menu needed at the login screen?

  1. If yes, how to enable left click&hold to open the contextual menu at login time?

  2. If no, should it not be nevertheless working at login screen for consistency reason? Or does in this case the fact prevail that a left click&hold is not a standard gui gesture and therefore disable it at the login screen. In any case, it should be implemented so to also work at login.

  The best way to resolve this is probably in combination with the accessible login feature. Here is the url of the corresponding spec: https://blueprints.launchpad.net/distros/ubuntu/+spec/access-gdm

MouseTweak #2:

• At the login screen, there should also be a way to activate dwell clicking; I think again that this question is better suited in the specification about an accessible login screen: https://blueprints.launchpad.net/distros/ubuntu/+spec/access-gdm

• What is better: a checkbox to automatically enable dwelling after login or making the account keep its activated/deactivated state across logins.

MouseTweak #3:

• Personally, I don't know how the tremor reduction works. Does it have to differentiate between smoothing a moving mouse and a motionless mouse. I assumed the former situation in my sample Mouse Preferences Tremor Tab above. But it is merely an assumption.

MouseTweak #4:

• Maybe we should drop the Dialogue entry in the list of applications for simplicity reasons and simply add the corresponding actions to global. In this case however, the user has to pay attention not to override in a specific application a gesture that he defined for dialogues in global.

BoF agenda and discussion

MouseTweak #1:

MouseTweak #2:

MouseTweak #3:

MouseTweak #4:

Comments

MouseTweak #1:

• The activation of the left click&hold to open the contextual menu conflicts with dwell clicking with a window to choose click type. (See Textmark A in the design section of dwell clicking for the point of conflict). So the user should be told about the conflict so to have the opportunity to choose which to activate.

• Depending on the timings set in mousetweak#1 and mousetweak#4,there might be a conflict. See first comment for mousetweak#4.

MouseTweak #2:

• The Human Interface Guidelines (HIG) 2.0 of Gnome also define copy and transfer actions for the middle mousebutton in combination with modifier-keys. I don't think there is a point to make dwelling emulate the middle mouse button for several reasons:
  • The dwell user can do the copy and transfer actions by using the cut copy paste commands of the Edit menu or the contextual menu.
  • I don't think that a middle mouse button added to the dwelling with the help of a window is more effective than using the Edit menu, because the click that is necessary to choose the middle click type can be used to open the Edit (contextual) menu.
  • When dwelling with gestures, the addition of the middle button click could enhance the effectivness, as we could define a stroke for these actions. But it would come at the expense of simplicity: At the moment, the is a click type for the 4 direction of movement (left, down, right, up). To introduce another button click we would have to add oblique directions (for example northeast) or introduce strokes made up of 2 direction executed one after the other (for example a move down followed by a move left).

  • The HIG 2.0 forbids to attach an action to the middle button that cannot be done in another way. Here is the corresponding quote from the HIG 2.0:If you do intend to use the middle button for a different purpose somewhere, only do so as a shortcut for experienced users, and only for operations that can also be performed without using the right button or middle button. Consequently, the dwell user probably also has a way to do every action as he can emulate the left click, left double click, right click and drag&drop.

• The activation of dwelling with a window to select clicktype should deactivate the left click&hold to open the contextual menu because there is a conflict between them. In fact doing a drag&drop without moving the mouse between the two dwell delays could be interpreted as a left click&hold, because it is in fact an emulation of a long mouse button down. (See Textmark A in the design section of dwell clicking). Anyhow, it does not make sense for a dwell user to activate mousetweak#1, because normally he is not able to use a mouse button, and because the dwelling software provides its own way to emulate a right button click.

• Mouse Gestures in timed mode can can be configured to provide dwell clicking with gestures and more. (see mousetweak #4)

MouseTweak #3:

MouseTweak #4:

• The mouse gestures software has to make sure that the time set in "Show contextual menu after" of mousetweak#1 is greater than the time set in "Click normally if there is no movement in:" of mousetweak#4. Otherwise the contextual menu opens while the mouse gesture software is still waiting for a mouse gesture.
• Mouse Gestures in timed mode can be configured to provide similar functions (even more functions) as those defined in dwelling with gestures in mousetweak #2.

---

CategorySpec