Libinput - alternate acceleration profile for touchpad

I don't like the libinput acceleration profile for my laptop touchpad.

I have Debian 11 (bullseye) on my laptop with xfce4 desktop. It defaults to libinput for the touchpad but I didn't like the acceleration profile. It was either too fast for accurate fine movements or too slow for traversing the display. While I could adjust the speed, I couldn't find any setting that was satisfying. I don't recall when the driver changed but it has been a long time now, maybe since I replace Mint with Debian a year or two ago. In any case, it is long enough that I am not merely suffering from a slight change to the acceleration profile conflicting with my muscle memory: even given plenty of time to adjust, I still don't like the libinput acceleration profile.

Recently, I decided to do something about it. I tried the Synaptics driver and it was better but still not what I wanted. I tried the evdev driver but it didn't work at all: the cursor didn't move no matter what I tried. No error message: just no movement. I really wanted to try some of the acceleration profiles that evdev provides but eventually I gave up.

It seems that the Synaptics and evdev drivers are deprecated and the libinput driver will predominate in the future, so I decided to focus on it. Others have criticized its acceleration profile, with sentiments like mine. The libinput FAQ includes:

Why is libinput’s pointer acceleration worse than synaptics/evdev

This is a known problem affecting some devices and/or use-case but the exact cause is still unknown. It may be a device-specific issue, it may be a bug in libinput’s acceleration code, it may be a disagreement about how pointer acceleration should feel. Unfortunately this is something that affected users need to investigate and analyze.

As I had tried all sorts of settings and given it lots of time to get used to it but was still bothered by it, I decided to try implementing an alternate acceleration profile. I am not hopeful that this will ever be accepted into libinput. For good reason, the developers need to keep it simple and robust with minimal support resources. Adding features adds support burden. But I only care about support for me and my one system, and it is, after all, free software.

It took me a couple of days to learn enough to get started but once I familiarized with the libinput code a bit, it turned out to be easy to implement an alternate acceleration profile with hard coded parameters: just one function to change. Adding configuration parameters is a bigger challenge for another day. But as I am recompiling libinput anyway, changing the parameters in the code isn't a problem, and I don't want to be changing them anyway, once I get them right.

I built libinput per Building. This gave me a local repository.

I created a new branch: wip/alternate_profile.

I modified src/filter-touchpad.c, which contains the 'adaptive' profile acceleration code. I created a new function, touchpad_accel_profile_constrained_linear, modeled after the original touchpad_accel_profile_linear (which is, in fact, not a linear transfer function) that implements the adaptive profile (the default profile) and changed the pointer to the transfer function to the new one. I don't yet know how to add the new profile selectable by configuration but I don't want to use the adaptive profile so replacing it is OK for now. It's a crude hack, but fixes my immediate problem and is a step towards adding a new profile.

The new profile has four hard-coded configuration parameters:

  const double minimum_factor = 0.05; /* unitless */
  const double maximum_factor = 0.75; /* unitless */
  const double lower_threshold = 15.0; /* mm/s */
  const double upper_threshold = 100.0; /* mm/s */

The acceleration factor is a simple function of speed:

   accel
   factor
     ^
     |       --------
     |      /
     |     /
     |----/
     +-------------> speed in

At speeds below lower_threshold, factor is minimum_factor.

At speeds above upper_threshold, factor is maximum_factor.

At speeds between the two thresholds, factor is a linear ramp from minimum_factor to maximum_factor.

This gives me precise control at slow speeds and a gradual transition to rapid movement at high speed. The transition is sufficiently smooth that it feels predictable. I may be wrong, but I think the linear ramp is fine: no need for any fancy curve, as long as the transition from slow to fast isn't too abrupt, i don't think the details of the curve matter. If it were a mouse, I might care about the details of the curve more, but a touchpad is so crude, it doesn't matter.

Otherwise, there are no 'magic' numbers, as there are in touchpad_accel_profile_linear, but there may be elsewhere. I haven't reviewed all the code of libinput, and much less so the Xorg libinput driver or Xorg server themselves.

The parameters are hard coded. There is no provision to set them by Xorg configuration files, xinput, xset or whatever. I don't yet know how to add such capability.

But it is easy enough to recompile and reinstall, so not too difficult to change the parameters.

Of course, installing this modified libinput affects the entire system and there is no provision for per-user configuration at this point. I am using it on my personal laptop, so there are no other users.

I am using this on an old Toshiba Satellite Pro C665. I also added a hwdb entry for it, as the default touchpad dimensions were incorrect. In  /etc/udev/hwdb.d/61-evdev-local.hwdb:

# Toshiba Sattelite Pro C665
evdev:name:SynPS/2 Synaptics TouchPad:dmi:*svnTOSHIBA:*pnSatelliteProC665**
 EVDEV_ABS_00=1122:5863:58
 EVDEV_ABS_01=951:5118:99
 EVDEV_ABS_35=1122:5863:58
 EVDEV_ABS_36=951:5118:99

I followed the instructions at Coordinate ranges for absolute axes. I was surprised to find that hwdb only had entries for two Toshiba systems. You might check your own touchpad to make sure it is correct.

If you are interested, you can clone it from wip/alternate_profile

To build and try the new profile:

1. $ git clone https://github.com/ig3/libinput.git
2. $ cd libinput
3. $ git checkout --track  origin/wip/alternate_profile
   
4. $ meson --prefix=/usr builddir/
5. $ ninja -C builddir/
6. $ sudo ninja -C builddir/ install
7. restart your X server (e.g. logout and login or reboot)
   

I have been using it for a few days now and it feels much more comfortable than the libinput 'adaptive' profile. I have good control at slow speeds and I can easily move the cursor across the screen. In summary, it works as I expected and I like it.

Linux touchpads in Xorg X Server with libinput

I have Debian 11 (bullseye) on my Toshiba Satellite Pro C665 laptop. It uses libinput by default but I found the cursor difficult to control. With a low acceleration I had precise control but it was tedious traversing the screen. With a high acceleration I could traverse the screen OK but precise control of the cursor was difficult to impossible. So, I began to learn far more than I ever wanted to know about Xorg X Server, libinput, synaptic, evdev and pointers in general. I have only learned a little: not yet enough to achieve reasonable behaviour.

My touchpad works with the libinput and synaptic input drivers but doesn't work with evdev. For reasons unknown, when I configure evdev there are no errors logged but the cursor doesn't move. The buttons are recognized but I can't move the cursor. Other's report this behaviour and the most common 'solution' is to use synaptic or libinput drivers. The events are generated. I have no idea why they are ignored when I try the evdev input driver. Maybe evdev is essentially unsupported software recent years and broke at some point and no one cares.

This thread addresses another fault in libinput but it includes instructions for re-building libinput from source. I have now done this both via 'apt build-dep' etc. as indicated there, and also from xf86-input-libinput which I built and installed with no ill effect.

The person who responded with the instructions for for building libinput said, among other things: "I agree that this isn't an ideal user experience. On the other hand, recompiling libinput in 2020 takes about as much time as taking out a credit card from your wallet and paying for another closed source macOS app." and there was some subsequent back and forth among respondents about how reasonable this workaround is.

I have been aware of X Server since the 1980's. I have been using Linux since the mid 90's, written device drivers from scratch, hacked parts of the kernel and built and debugged many packages through the years. I have been searching for a way to improve my touchpad behaviour for two days now and have learned only a little of what software is involved, how it all relates and how it is all configured. Had I not stumbled on the above linked thread, it might have been many more days before I learned how to build libinput from source and much longer to understand what all the code was doing to resolve that problem. I have looked at the code that implements the acceleration profile but it will be a long while before I understand how to add my own profile and configuration options (i.e. changes to processing of the Xorg configuration files, xinput, xset and who knows what other places might be required to make something that works).

The documentation (in all forms I can find, including man pages, tutorials and forum discussions) is typically terse, often using undefined terms and often inconsistent with the current implementation on my laptop. The software and how to configure it has obviously changed through the years and it is difficult to determine what advice applies to what version of the software. Even current Xorg documentation on the official Xorg website is inconsistent with my experience when using the libinput device driver.

My conclusions thus far:

The evdev device driver doesn't work: there are no errors but the cursor doesn't move when I use the touchpad. No idea why. Much of the documentation I can find about setting acceleration profiles seems to be specific to the evdev input device driver. I would like to try some of these profiles but I can't because the cursor doesn't move at all.

The synaptic device driver works but it is difficult to understand how to configure acceleration. Synaptics Pointer Acceleration makes some convincing arguments that the Synaptic driver is complex to the point of being unpredictable, though it might not be so intractable if one cares only about a single hardware configuration. The author is one of the main contributors to both the Synaptic and libinput device drivers, so well informed. After some effort I was unable to achieve what I wanted with this input driver.

The libinput driver is the default on my and many systems. Many complain about the acceleration. Documentation of it is misleading and there are very limited configuration options. I have been unable to configure it to allow my both fine control of the cursor and easy traversal of larger areas. The source code suggests (to my as-yet superficial understanding) that the implementation is more complex than and inconsistent with the documentation, with obscure heuristics determining the acceleration parameters, rather than configuration parameters.

It is a dismaying prospect to have to spend many more days if I want to understand what all the configuration options (across GUI configuration tools, command line tools and various configuration files for Xorg, udev, xfce4, etc., etc., etc.) are and what they do and which part(s) of the software they affect.

Presumably there is code in the hardware device driver, Xorg input driver and Xorg X Server that could and possibly do contribute to the overall mapping from the hardware to the cursor position. But it is a morass of poorly documented components, leading to confusion and every experiment thus far requires an hour or two of researching errors, installing and configuring tools, often to achieve little to nothing.

Spaced Repetition Flashcards

After two years using Anki, I decided to write my own spaced repetition flashcards software.

I was using the Anki v2 scheduler but it had various faults and limitations, the most serious of which were a fixed number of new cards per day, lack of API to implement an alternate scheduler, frequently changing add-on API and overly complex build environment that requires familiarity with several languages and build tools. 

I created add-ons that overcame some of the faults but it became too difficult to maintain them. And they didn't achieve all that I wanted. Because of how many of the scheduling features are implemented, many problematic aspects are difficult or practically impossible to alter or improve. It became evident that it would be less work and better outcome to write my own software than to continue struggling with Anki.

I use only a small subset of Anki features. Only an single instance of the desktop version (so no syncing) and only simple cards - no cloze cards. The Anki application was vastly more complicated than I needed.

So, I wrote my own. It is browser based with the server running on nodejs. It is simple: less than 1000 lines of JavaScript and a few templates. It's a bit crude yet, but I have been using it to study for over a week now. It presents all my cards as well as Anki did. It has, in my opinion, superior scheduling. The build tools are simple and well known - in other words: easy to install and use. I only need to know one programming language: JavaScript, or five if you include the SQL, Handlebars templates, HTML and CSS as languages. All but the Handlebars templates are ubiquitous and familiar, and Handlebars templates sufficient for this purpose, are simple.

About the same time, I updated my Anki add-on again, to support up to the then current 2.1.43 (and probably a few subsequent - until the internals change again). This is likely to be the last time I will update the add-on as I am not using it or Anki any more. 

Scheduling is simpler and more consistent than Anki. New cards are automatically regulated to maintain a steady workload of about 1 to 2 hours of study per day. The number of cards to be reviewed each day is somewhat random, but if study time in a day (actual or projected) exceeds 1 hour then no new cards are presented. This allows me to focus on the cards I am already learning, until I learn them well enough that cards to be studied / study time fall below the threshold, after which new cards are presented again. Other than the limit on total study time, there is no limit on new cards.

The scheduler prioritizes cards with shorter intervals over cards with longer intervals. This doesn't make much difference when you are up to date but it makes a big difference with a backlog, as I had after working on on this new app for a few days instead of studying. Even with a large backlog, the challenging cards don't get deferred by the backlog. It seems, in this way, to be much better than Anki. A few days after I got back to studying, I had cleared my backlog and was seeing more new cards than I had in a long time with Anki.

I have no research that proves my scheduler is more effective than Anki's but at least it doesn't have the misfeatures that I experienced with Anki: fixed number of new cards per day; excessive complexity of new, learning, re-learning and review cards and queues; a tendency to gross overload and dysfunctional review ordering, resulting in overall failure to learn; adverse interactions of new card limits between nested decks; lack of control of new card order; and limitation of the scheduling algorithm to units of days.

It is based on an import and modification of my old Anki database. I haven't yet written anything to import an Anki (or other) shared deck, but I probably will. However, currently I have about 30,000 new cards, so no need for more any time soon. I wrote an utility to import the Anki database but probably won't use it again: it would be too much of a setback to revert to what I was doing in Anki weeks ago and I don't study with Anki any more. I suppose I might import a new database if I used Anki to download some other decks and wanted to keep multiple databases, but it would probably be better to write an import of shared decks directly than to do it through Anki.

There is a very crude interface for editing notes: just the fields. So, I can correct errors as I find them. I'll add features to add and remove notes at some point.

New card ordering isn't lost when a card is studied. As a result, it is possible to reset a deck to it's original state and start over with the original ordering of new cards. This isn't possible with Anki because the initial ordering is in the due field of the cards, which is overwritten when the card is viewed.

Some of the serialized data in the Anki database is no longer used but there is more to be done. There is nothing fundamentally wrong about storing serialized data structures, but using a non-standard format is bad. I might revise some of the serializations to JSON. Alternatively, I might deserialize them into separate fields in the database. But, for the moment, I have cracked the rust serialization algorithm, so there is no immediate need.

There is room for improvement in the review log, but I can produce basic statistics such as counts of cards studied and due per day, study time per day, number of new cards viewed per day, etc.

Learning a Language with Anki and other resources

 I like what this reponse to this post says about words Vs sentences for learning a language, and the use of Anki Vs other resources. And this post expresses the same ideas about context Vs isolated words, though I am more inclined to somewhat longer, more complex sentences than they are. When learning a language, I am not concerned about remembering the idea expressed by a sentence. I am only concerned with learning to understand the sentence, to extract the meaning. The issue of complexity might be much more significant if one were trying to learn the expressed ideas (e.g. math, physics, economics or whatever) rather than learning to understand the expression.

Anki Scheduling

I have been using Anki on Linux to help my study of Mandarin. I am just a beginner, learning basic vocabulary.

Anki has been very helpful but after a few months of study, with default scheduling, I became overwhelmed. It was taking me several hours to complete all scheduled reviews and many days I didn't have enough time. I was seeing cards too infrequently and not learning effectively. I was failing. Something had to change.

I'm not the only one. Many others have faced this problem and much has been written. Search for Anki Scheduler and you can find much more, and many good ideas.

For me, the fundamental problem was that I had too many cards to learn and review, and new cards were being added every day - faster than I could learn them. Everyone has limits. My memory is poor so my limit is quite low. I had to reduce the number of new cards I was seeing.

I was studying three main decks: a large deck of characters, a large deck of phrases and a growing deck of my own - characters and phrases that I had seen in my other study (reading and watching movies and serials). I merged the latter two into a single deck and then put the remaining two decks under a single parent deck. This gave me one deck (the parent deck) to study each day.

I set new cards per day low but it wasn't enough. I was so overwhelmed, I wasn't making progress, even with a low number of new cards per day. I realized I had to stop the new cards and focus on the cards I was already learning, at least until I had recovered.

I could have set new cards limit to 0 until I recovered, then either increase the limit or add new cards through custom study, but I didn't want to have to make such decisions every day.

I could have set new cards to show only after I had completed all scheduled reviews. If I didn't complete scheduled reviews I would see no new cards. It would be self-regulating. But it was difficult to learn new cards after a long period of study and not completing all that was scheduled, including new cards, felt too much like failure.

I wanted new cards to be mixed with review cards but I didn't want too many new cards, and I didn't want to be adjusting the new card limit manually. I was using the new version 2 scheduler but there was no way to achieve this except by manually adjusting the new card limit up and down. I wanted something more automatic. Something that challenged me but didn't overwhelm me.

I wanted to study between one and two hours per day. If I wasn't able to complete all scheduled reviews in that time, I didn't want to add any new cards. On the other hand, if I was completing scheduled reviews on time I wanted to see new cards, to make progress and continue to be challenged. I wanted to be studying near my limits but not go beyond them.

The scheduler couldn't do this for me, but Anki is extensible, so I wrote an add-on to limit new cards. The add-on limits the number of new cards based on the number of reviews scheduled and completed. It allows me to mix new cards with reviews, to reduce the number of new cards when I am overwhelmed but see enough new cards to challenge me when I am not. It keeps me near my target of one to two hours of study per day. It is not all-or-nothing. As number of cards to study increases, number of new cards decreases gradually. Even when I am near my limit, I see a few new cards but if I am overwhelmed, there are no new cards until I recover.

The add-on is based on number of card views rather than time, but that's OK. Time per card is fairly consistent and it's only an approximation to my capacity in either case. I configured it to begin limiting new cards when scheduled reviews exceeds 150, with no new cards at all if scheduled reviews exceeds 250.

It worked as expected. Initially, because I had such a large backlog of unlearned cars, I saw no new cards at all. Gradually I learned the cards I was already studying. After a few weeks, I was consistently completing all scheduled reviews. My daily study time came down, so I wasn't exhausted. I began to learn more effectively again. As my number of daily reviews came down, I began to see new cards again. It was success.

Now that I am through the worst of the overload, my daily study time is usually between 1.5 and 2 hours. On a bad day (if I have had no sleep) sometimes more. On a really good day, only a little over an hour. This fits my overall schedule. If I miss a day or two, it takes me a few days to catch up. While I am catching up, I see no new cards, but when all is going well I see a few new cards each day - more or less, depending on my workload, keeping me near my configured limit of about 200 reviews per day.

It is still early days. I expect it will take me many years to learn enough Mandarin to have conversations, watch movies without subtitles and read news and novels. But now, I learn a little every day, and my time with Anki is effective. I see enough new cards to make progress and keep me challenged, but not too many. I am no longer overwhelmed.

Everyone is different, learning differently, with different capacity and limits. What works for me (I have poor memory, getting worse as I get older) may seem trivially easy for others. But Anki is configurable and each user will have to find the configuration that suits their ability and interest. The default configuration is a good starting point, but I have made adjustments.

To understand the configuration, you have to understand the Anki scheduling algorithm. Read the manual for a basic introduction to the options and the FAQ on the Anki Algoritm. I also found this video helpful.

Consolidating decks helped. I am only studying Mandarin. If I were studying different subjects I would probably keep them in separate decks, but for Mandarin, I prefer characters and phrases to be mixed, rather than separate.

As much time with other study (watching movies and reading) as I spend with Anki also helps. My ears need practice and the repetition and seeing things in different contexts really helps.

Currently, I have new cards per day limited to 5. My memory really is poor. I have little capacity. I could increase this, as the add-on limits new cards. I haven't seen 5 new cards in a day for a long time now, but I am still working through the backlog I accumulated before writing the add-on. It will probably take me a few more months to reach a stead state of learning. Maybe then I will increase the limit on new cards. If not, I can live with 5 new cards per day. Slow and steady is the best I can do these days.

For new cards, I have Graduating interval at 1 days, Easy interval at 4 days and starting ease 175%. Initial steps are 1, 2, 5, 10 and 20 minutes. I need lots of repetition to learn but after the initial repetitions, I prefer to let the Anki scheduler adjust the intervals, rather than a fixed schedule. Some cards are easier and require fewer, less frequent repetitions. Others, not obviously different, I find difficult to learn - they require a lot of frequent repetition. The scheduler takes care of this for me.

For reviews, I have a maximum of 1000 per day. I want to complete all reviews on schedule. I don't want them delayed by this limit. Easy bonus is the default 150%. Interval modifier is default 100%. But I decreased the Hard interval to 80%. If I find a card hard, I want to see it sooner. If it continues hard, I want the interval to decrease until I am seeing it often enough that I start to remember. I really do need a lot of repetition. 

For lapses, I have steps 3, 5, 10 and 60 minutes - a little burst of frequent repetitions. Then, New interval at 30%. I want to see the card again soon. I'm not concerned about leeches, so I set leech threshold to 50 lapses and Leech action to Tag Only.

With these settings, I am making progress. But I have noticed a few faults / features of the V2 scheduler that I don't like. So, I wrote some more add-ons to fix/improve these.

With Hard interval set above 100% and low ease, the minimum increment in interval is 2 days. Cards progress through 1 day, 3 days, 5 days etc. This rapid progression is sometimes too much for me - it is effectively a minimum ease of 300%, only decreasing to the card's actual ease when the interval has reached a week or more. I am good for one or two reviews then lapse again. Cards end up with minimal ease (Hard and Again both decrease ease) but still progress too quickly.

Worse, with Hard interval set below 100% and low ease (anything below 150% - and minimal ease is only 130% and Hard and Again both reduce ease) they get stuck at an interval of 2 days - not progressing at all. Easy will lift them out of this, but Good leaves them stuck at 2 days indefinitely. And I want Hard interval at 80% or 90% - if I find a card hard, I want to see it more frequently, not less.

So I wrote this add-on to fix the calculation of new intervals for review cards. It fixes both problems. Regardless of Hard interval, minimum increment in interval is 1 day on Good. Not 2 days and never 0 days. If a card has an interval of 1 day, hard leaves it at 1 day (1 day is the minimum interval for a review card) and Good progresses it to 2 days. That's still an effective ease of 200%, but the minimum increment in interval and better than 300%. If a card has an interval of 2 days, Hard reduces it to 1 day (I have Hard interval at 80%) and Good increases it to 2 days. I was surprised, but I have noticed a significant improvement in my success with new and lapsed cards after making this change. I'm still working through a large number of cards with low interval and minimal ease - a bit of 'Ease Hell', but progress is better with this change. It seems I have an aversion to Easy, which would increase the ease, so progress is slow but at least it's steady.

Then I wrote this add-on to fix the fuzzed interval ranges.  This is a really trivial issue. Just something I noticed as I was watching my progression and the impact of the previous add-on. At low intervals, the fuzzed ranges were sometimes larger but sometimes smaller, even 0, as the interval increased. The add-on makes the fuzzed range increase monotonically with interval, with a logarithmic roll-off. The fuzzed interval range is about two weeks at an interval of one year. That seems plenty to ensure pairs of cards don't keep appearing on the same day.

Finally, I wrote this add-on to slightly increase ease on Good. I had read about 'Ease Hell' and with my backlog of cards with many lapses and many Hard cards, I had experienced it - many cards with low ease and low intervals, slowly progressing through Good, Good, Good. I really do have an aversion to Easy, unless a card is very, very easy. I could just choose Easy, but it seems more reasonable to me that if a card is Good, ease should increase a little and if it is Easy it increases a lot. This way, ease won't get stuck, every review changes it a little:

• Again: decrease ease by 20%
  
• Hard: decrease ease by 15%
  
• Good: increase ease by 5%
• Easy: increase ease by 15%

Note: ease is itself a percentage of the interval. For example, if ease is 200%, then on Good the interval is increased to approximately 200% of the current interval (i.e. the interval is doubled). The change is approximate because of interval 'fuzz'. The changes indicated above are percent of interval, not percent of ease. If ease is 200 percent, Good increases it to 205% and Easy increases it to 215%.

These changes are all hard coded in the original scheduler and this add-on. I might add configuration but the add-on is just a bit of python script so it is easy to change them if you want something different.

If a card continues Good through many reviews, ease will slowly increase. Eventually it should reach an ease/interval at which it becomes Hard, which will decrease the ease (and, if Hard interval is less than 100%, also decrease the interval, which I think is better than the default of increasing the interval to 120%). Ease should then go up and down as reviews alternate between Good and Hard - keeping interval near the limits of retention, which research suggests is good for learning. What it won't do is get stuck at an unreasonably low ease, with very slowly increasing interval as you keep finding the card Good.

Ideally, the card will never be Easy (i.e. Anki will progress interval fast enough that you are not wasting your time reviewing it too often) and never lapse (Again). It may be Hard, sometimes but will mostly be Good. That's the sweet spot of challenging your memory but not waiting so long you forget (lapse) and have to relearn the card.

It's early days for this add-on but I have many cards with minimal ease (130%) and I can now see them gradually increasing. My 'Ease Hell' is being alleviated. I can still use Easy to progress more rapidly, if I want to, but the ease and intervals will sort themselves if I persist with Good. And this is good!

While I haven't proved it with objective analysis of my study outcomes, collectively, these changes have been good. I am no longer overwhelmed, I am making progress and I am enjoying study again.

The first add-on is published to AnkiWeb. The others are relatively minor and recent, so I haven't published them yet, but if you are interested you can download them from GitHub and install them yourself. If there is interest, I'll add zip files to make the installation easier, or publish them to AnkiWeb. Let me know if you are interested but in the meantime, they are very easy to install manually.

I think Anki is great and with these few changes to the scheduling, it is even better for me. If you try it, I hope you too find it helpful and maybe my experience with it helps you.

Misc links:

• experimentalCardEaseFactor
• anki_algorithm.py
  

Building PouchDB on Windows 7

Building PouchDB on Windows 7 has been a rather miserable experience. Not because of PouchDB but because of the Microsoft Windows development tools: Visual Studio and Windows SKD.

For hours, every attempt to 'npm install' the PouchDB package failed with:

C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(29
7,5): warning MSB8003: Could not find WindowsSDKDir variable from the registry.
  TargetFrameworkVersion or PlatformToolset may be set to an invalid version nu
mber. [C:\Users\ian\Documents\Entrain\pouchdb\pouchdb-master\node_modules\level
\node_modules\leveldown\deps\leveldb\leveldb.vcxproj]
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Platforms\x64\Microsoft.Cpp.x
64.targets(146,5): error MSB6006: "CL.exe" exited with code -1073741515. [C:\Us
ers\ian\Documents\Entrain\pouchdb\pouchdb-master\node_modules\level\node_module
s\leveldown\deps\leveldb\leveldb.vcxproj]

The Internet is full of reports of this warning and this error. Most of them with only cryptic hints as to how to resolve them or nothing more than the comment "works for me". I spent over four hours installing, uninstalling, re-installing various packages from Microsoft and rebooting a mind-boggling number of times, following advice here and there. None of it made any difference. Always, the same warning and error.

Finally, I stumbled on https://github.com/node-xmpp/node-expat/issues/57 - an issue for node-expat. The last post there was from kosmobot:

@southrop you have to explicitly specify the Platform Toolset when building the solution with msbuild. Try one of these commands depending on your platform:
call "C:\Program Files\Microsoft SDKs\Windows\v7.1\bin\Setenv.cmd" /Release /x86
call "C:\Program Files\Microsoft SDKs\Windows\v7.1\bin\Setenv.cmd" /Release /x64

This actually solved the problem. So, I had all the prerequisites installed all along - the only problem was setting up the environment. Too bad none of Microsoft's replies to reports of the warning or error bother to mention this. If only I had found this post earlier, I might have done some useful work this morning. 

Much thanks to kosmobot!!

Magento object initialization (_construct Vs __construct)

Some classes in Magento define function _construct, some define __construct, some define both and some define neither. What's going on?

The function __construct is executed by PHP when an object is instantiated, if it exists, but function _construct is not. To PHP there is nothing special about _construct.

Initialization in Magento seems to be quite variable. Many classes define __construct. Many, but not all of these, execute parent::__construct - some before and some after local initialization.

In a few cases, __construct ends with $self->_construct(). The Varien_Object class is an example. In these cases, sub-classes can define _construct, in which case the parent's __construct will run and the sub-class's _construct will be run.

Class Varien_Object defines function _construct with the following comment:

    /**
     * Internal constructor not depended on params. Can be used for object initialization
     */

Only a few of the sub-classes of Varien_Object define function _construct. I haven't reviewed them exhaustively, but a significant majority of those I reviewed did not define function _construct.

Class Mage_Core_Block_Abstract also defines function __construct to execute $self->_construct(), and defines function _construct with the comment:

    /**
     * Internal constructor, that is called from real constructor
     *
     * Please override this one instead of overriding real __construct constructor
     *
     */

At least in this case there is a clear preference stated regarding __construct, but still not rationale.

On the other hand, at least one sub-class of Varien_Object (Mage_Core_Model_Email) defines function __construct and does not execute parent::__construct(). Nor does function __construct in class Mage_Core_Model_Email do the equivalent of function __construct in class Varien_Object.  So, the function __construct in class Varien_Object is not always executed in sub-classes of Varien_Object.

Sub-classes defining __construct and executing parent::__construct() seems to be the most common. It is hard to imagine any advantage of using function _construct(), as with Varien_Object. It has been suggested that it makes it less likely to forget to execute parent::__construct(), but this hardly seems likely. One would have to remember to define function _construct rather than function __construct. If one can remember this, then surely one can remember to execute parent::__construct(). It would be a more credible explanation if the use of function _construct were common: it might become habitual or one were not otherwise familiar with PHP. But in fact, classes using function _construct are uncommon. The vast majority of __construct functions do not execute $self->_construct().

Keep in mind that if a sub-class of a class that defines function __construct to execute $self->_construct() defines function _construct and a sub-class of that sub-class also defines function _construct, then when the root class executes $self->_construct, only that in the sub-sub-class will be executed: the function _construct of the intermediate class will not be executed, unless the sub-sub-class executes parent::_construct().
I haven't reviewed the classes of Magento exhaustively, but I have reviewed a significant number and thus far have seen only one (Mage_Core_Block_Template) where function _construct executes parent::_construct().

If sub-sub-classes are going to execute parent::_construct(), it seems to me that it would be simpler to simply use __construct and parent::construct() in the way that would be obvious to any PHP programmer, and do away with function _construct. But that's just my opinion.

In summary, in a few classes function __construct executes $self->_construct() and some sub-classes of these classes define function _construct rather than function __construct. I know neither the motivation nor the benefit of this, particularly as it is so uncommon. It is certainly not the norm in Magento. There has been some discussion on the Magento forums, but no explanation from a core team developer that I have seen.

Magento observer class

One of the parameters of a Magento observer is the 'class'.

For example, the Magento wiki has an example of customizing Magento using an observer. This example includes configuration of the module that provides the observer:

    <?xml version="1.0"?>
    <config>
      <global>
        <models>
            <xyzcatalog>
                 <class>Xyz_Catalog_Model</class>
            </xyzcatalog>
        </models>
        <events>
          <catalog_product_get_final_price>
            <observers>
              <xyz_catalog_price_observer>
                <type>singleton</type>
                <class>Xyz_Catalog_Model_Price_Observer</class>
                <method>apply_discount_percent</method>
              </xyz_catalog_price_observer>
            </observers>
          </catalog_product_get_final_price>     
        </events>
      </global>
    </config>

I wondered why, in all the examples I found, the observer class was always a 'Model' class. It seems strange to me that a 'Model' (think MVC) would handle an event. I think of a Model as getting or persisting data. A controller seems a more appropriate component for handling an event. But, again, every example I have seen executes a method from a Model class. So, I had a look at the Magento code to see what was going on and whether there were any clues as to why a Model rather than a Controller.

In some cases, like the example above, the class name is given explicitly. In every case I have seen, the class name includes 'Model'. Note that in a case like this example, the Mage __autoload function changes '_' to directory separator so, on a Linux system, that class would be loaded from Xyz/Catalog/Model/Price/Observer.php. In this case, it is just a class and needn't be a Model as far as I can tell.

In other cases, the class is specified differently: as 'module/model'. In this case, the processing within Magento inserts 'Model' into the class name, so it is a bit more explicitly a Model class. See getGroupedClassName in Mage_Core_Model_Config for the full details. getGroupedClassName is called to transform the class name if it contains '/', in which case the class becomes getGroupedClassName('model', $class).

Magento observer types

When configuring an Observer in Magento, one of the configuration parameters is 'type'.

For example, the Magento wiki has an example of customizing Magento using an event observer, where the module configuration is:

<?xml version="1.0"?>
    <config>
      <global>
        <models>
            <xyzcatalog>
                 <class>Xyz_Catalog_Model</class>
            </xyzcatalog>
        </models>
        <events>
          <catalog_product_get_final_price>
            <observers>
              <xyz_catalog_price_observer>
                <type>singleton</type>
                <class>Xyz_Catalog_Model_Price_Observer</class>
                <method>apply_discount_percent</method>
              </xyz_catalog_price_observer>
            </observers>
          </catalog_product_get_final_price>     
        </events>
      </global>
    </config>

Note the 'type' key in the configuration. In this case, the content is 'singleton', but there is no explanation of what this aspect of the configuration is about.

The type is dealt with in the dispatchEvent method of class Mage_Core_Model_App, method dispatchEvent, which is executed from class Mage, method dispatchEvent (the latter seems to be what is executed generally throughout the code but it is just a thin wrapper around the former). This function ends with a loop that executes each observer registered for the event in turn, as follows:

            foreach ($events[$eventName]['observers'] as $obsName=>$obs) {
                $observer->setData(array('event'=>$event));
                Varien_Profiler::start('OBSERVER: '.$obsName);
                switch ($obs['type']) {
                    case 'disabled':
                        break;
                    case 'object':
                    case 'model':
                        $method = $obs['method'];
                        $observer->addData($args);
                        $object = Mage::getModel($obs['model']);
                        $this->_callObserverMethod($object, $method, $observer);
                        break;
                    default:
                        $method = $obs['method'];
                        $observer->addData($args);
                        $object = Mage::getSingleton($obs['model']);
                        $this->_callObserverMethod($object, $method, $observer);
                        break;
                }
                Varien_Profiler::stop('OBSERVER: '.$obsName);
            }

Note that there are really only three cases for type: 'disabled', in which case no observer is called; 'object' or 'model', which are equivalent, in which case Mage::get_Model is called; or any other values (any other value is equivalent - this includes 'singleton'), in which case Mage::getSingleton is called. Mage::get_Model and Mage::getSingleton are both passed the value of the 'model' parameter of the observer configuration.

Mage::getSingleton($class) calls Mage::getModel($class) but it caches the return value and calls Mage:;getModel only once. Subsequent calls for the same $class return the cached instance rather than a new instance. In contract, Mage::getModel($class) returns a new instance of the class every time.

So, for new modules, one might use the types: 'disabled', 'model' or 'singleton'.

Type 'disabled': the observer class is not instantiated and the observer method is not executed.

Type 'model': a new instance of the observer class is instantiated for each event and the observer method of that instance is executed.

Type 'singleton': a single instance of the observer class is instantiated and the observer method of that single instance is executed for each event.

Magento installation layout

I inherited Magento in a BitNami image. These are just my notes as I familiarize myself with Magento - probably not very reliable as I don't know much about it yet. Judging by the RELEASE_NOTES.txt file in htdocs, it is version 1.7.0.2. This is the 'Full Release' version of Community Edition currently available from the Magento Commerce download page. I don't know about the BitNami image version, but it too is quite recent.

Magento is installed to /opt/bitnami/apps/magento. It is accessed via an Apache virtual server, configured in /opt/bitnami/apache2/conf. The document root folder is /opt/bitnami/apps/magento/htdocs. Almost the entire Magento installation is within the htdocs folder.

Within /opt/bitnami/apps/magento there are:

• conf (folder)
• htdocs (folder)
• licenses (folder)
• scripts (folder)
• updateip.backup (file)

The conf folder contains three files. One relates to Beetailer, which I assume relates to this Facebook integration software. Another is a bit of Apache configuration, which is included into the main Apache configuration in /opt/bitnami/apache2/conf/httpd.conf: this sets up aliases and directory permissions for the Magento installation but does not define the virtual website. Note that the permissions here are wide open for the htdocs folder, but there are .htaccess files that restrict permissions, so it is not a bad as it looks initially. Finally, there is magento_info.txt which contains an encryption key. This may be from BitName. It is vaguely described in their FAQ.

The htdocs folder contains the bulk of the Magento installation. More on that later.

The license folder just has a copy of the license under which the Magento Community Edition software is released.

The scripts folder contains a single sql script (beetailer.sql) which appears to be related to setup of the Beetailer software noted above.

Finally, updateip.backup. This file appears to be part of the BitNami image. It is renamed so that it doesn't run. According to the BitNami Magento FAQ, it is for updating the Magento configuration with the server IP at system boot. Advice for WordPress is to rename it, appending '.backup' to the name, if the server has already been configured. So, it seems a reasonably safe assumption that this file is not doing anything. It is an ELF executable, none the less.

The htdocs folder contains the installation of Magento, almost in its entirety.

Perhaps a good place to begin is with htdocs/.htaccess. This file limits access to the folder, which otherwise is wide open, and provides configuration for various Apache modules. Among other things, it defines the default directory index as 'index.php'.

There are five main php scripts:

1. api.php
2. cron.php
3. get.php
4. index.php
5. install.php

The api.php script handles requests to the Magento API. It is part of the Mage_api2 package. I don't know anything about this yet.

The cron.php script handles routine processing. I understand this should be executed on a regular schedule (perhaps daily???) and, possibly among other things, is needed to maintain the Magento cache.

The get.php script appears to be for downloading resources. Again, I don't know anything about this, but at a quick read it looks like it handles returning resources either from the file system or the database.

The index.php script is the main Magento entry point. It handles routine access to the Magento site / store. This script itself is quite brief. It does a little initialization, loads the main Mage script and executes Mage::run().

The install.php script is part of an installation option. Rather than copy the entire Magento installation to the server, it is possible to copy a small subset, including install.php, the downloader folder, which contains code supporting download and installation of Magento, and, no doubt, some other bits of configuration (contained in a relatively small zip file available for download, I believe), then access install.php from the browser to downlaod and install the full Magento release, assuming the web server has sufficient permissions on the htdocs folder.

The app folder contains most of the Magento 'application'.

The downloader folder contains script and configuration for downloading and installing Magento, supporting the install.php script.

The errors folder contains script and resources for handling errors and returning error messages.

The includes folder contains just one file: config.php. In a default install, config.php does nothing. It has lines to set PHP named constants COMPILER_INCLUDE_PATH and COMPILER_COLLECT_PATH, but these are commented out.This relates to the Magento compiler, which can be used to assemble all the class files into a single folder. This compiler is described by Alan Storm, who has many excellent posts about Magento - well worth reading.

The js folder is misleading. It does contain a little javascript, but it also contains css files, various image files and more. I guess it contains the client side parts of the Magento application.

The lib folder contains various libraries and resources used by the Magento application (or constituting the Magento application, depending on how you look at it).

The media folder contains some image files, xml files and various cryptically named files. No idea about this yet.

The nbproject folder contains very little. There are references to Net Beans. No idea about this...

The pkginfo folder contains some 'metapackage' information files. Part of how Magento is packaged and distributed, I imagine.

The shell folder contains a few php scripts that are part of the Mage_Shell package, whatever that is.

The skin folder contains more css and image files. Presumably these are related to styling the Magento interface.

The var folder contains various more or less temporary files, including the Magento cache. Presumably this is mostly managed by the Magento application.

Otherwise, there are license files, release notes and a robots.txt file.

Magento observer method arguments - what are they?

One of the difficulties I have developing event observers for Magento is that I don't know what the arguments to the observer methods are. Most examples show a single argument, typically named $observer. What is it and what are its methods and attributes? I have had difficulty finding this out.

Initially I dumped the argument to a log using print_r:

        public function observer_method($observer) {
            Mage::log(
                "Observer observer_method executing with: " .
                print_r($observer,true),
                null,
                'MyModule.log'
            );
        }

This worked fine for a while, then I tried this with the argument passed for the sales_order_item_after_save event and quickly ran out of memory. The problem is that the passed object has cyclical links and print_r doesn't notice the recursion: it just keeps printing until it runs out of memory. The var_dump function has the same problem.

Fortunately, most objects in Magento are derived from the Varien_Object class and this class has a debug() method which handles recursion. So, a more general solution to inspecting data in Magento is to combine print_r with debug:

        public function observer_method($observer) {
            Mage::log(
                "Observer observer_method executing with: " .
                print_r($observer->debug(),true),
                null,
                'MyModule.log'
            );
        }

The debug method returns an array with no recursion and print_r renders that to a string.

Another approach is to find where Mage::dispatchEvent is executed for the event of interest and examine the arguments that are passed. This is easy for some events: Mage::dispatchEvent is called with the event name as a literal argument. One can grep the source for these. But, again, the case of sales_order_item_after_save was more challenging. Grepping the source for this event yielded nothing except observers. Eventually I grepped for 'sales_order_item' and 'after_save' separately and found where the event might be dispatched...

In app/code/core/Mage/Core/Model/Abstract.php one finds:

    /**
     * Processing object after save data
     *
     * @return Mage_Core_Model_Abstract
     */
    protected function _afterSave()
    {
        $this->cleanModelCache();
        Mage::dispatchEvent('model_save_after', array('object'=>$this));
        Mage::dispatchEvent($this->_eventPrefix.'_save_after', $this->_getEventData());
        return $this;
    }

So, since I can't find it elsewhere, I'm guessing whatever issues the sales_order_item_save_after event is calling _afterSave() with _eventPrefix set to 'sales_order_item'.

I found one class (Mage_Sales_Model_Order_Item in app/code/core/Mage/Sales/Model/Order/Item.php) that extends Mage_Core_Model_Abstract and sets a property _eventPrefix to 'sales_order_item'. It doesn't call _afterSave itself, so I still don't know exactly what the passed arguments are, but getting closer. The Mage_Sales_Model_Order_Item class doesn't have a _getEventData() method, so it is most likely that _getEventData from class Mage_Core_Model_Abstract is the culprit.

From Mage_Core_Model_Abstract:

    /**
     * Get array of objects transfered to default events processing
     *
     * @return array
     */
    protected function _getEventData()
    {
        return array(
            'data_object'       => $this,
            $this->_eventObject => $this,
        );
    }

and

    /**
     * Parameter name in event
     *
     * In observe method you can use $observer->getEvent()->getObject() in this case
     *
     * @var string
     */
    protected $_eventObject = 'object';




But, class Mage_Sales_Model_Order_Item has:

    protected $_eventObject = 'item';

So, the argument to the event observer for sales_order_item_save_after should be an array with two elements: 'data_object' and 'item', but both referring to the same data: the Mage_Sales_Model_Order_Item instance.

Maybe next time I'll try generating a stack trace in the observer method. That should help to pin down the method that dispatches the event quickly.

jquery 1.7.1 and IE8

jQuery 1.7.1 running on IE8 fails when trying to call the focus method on elements while opening jQuery UI dialog.

In  jquery-1.7.1.js, function jQuery.event.trigger (a method of jQuery.event) begins with:

    trigger: function( event, data, elem, onlyHandlers ) {

        // Don't do events on text and comment nodes

        if ( elem && (elem.nodeType === 3 || elem.nodeType === 8) ) {

            return;

        }

and ends with

            // Call a native DOM method on the target with the same name name as the event.
            // Can't use an .isFunction() check here because IE6/7 fails that test.
            // Don't do default actions on window, that's where global variables be (#6170)
            // IE<9 dies on focus/blur to hidden element (#1486)
            if ( ontype && elem[ type ] && ((type !== "focus" && type !== "blur") || event.target.offsetWidth !== 0) && !jQuery.isWindow( elem ) ) {
                // Don't re-trigger an onFOO event when we call its FOO() method
                old = elem[ ontype ];

                if ( old ) {
                    elem[ ontype ] = null;
                }

                // Prevent re-triggering of the same event, since we already bubbled it above
                jQuery.event.triggered = type;
                elem[ type ]();
                jQuery.event.triggered = undefined;

                if ( old ) {
                    elem[ ontype ] = old;
                }
            }
        }
    }
    return event.result;
},

In the last bit, change

elem[ type ]();

to

try {
     elem[ type ]();
} catch(err) {
     // ignore errors
}

 With this change the dialog opens successfully.