Kevin Kenny$Revision: 1.13 $

Most popular programming languages provide some sort of indexed array construct, where array subscripts are integers. Tcl's lists are implemented internally as indexed arrays, but it is difficult to use them as such because there is no convenient way to assign to individual elements. This TIP proposes a new command, lset, to rectify this limitation.

The implementation of lists in Tcl has evolved far beyond the original conception. While lists were originally conceived to be strings with a particular syntax that allowed them to be parsed as lists, the internal representation of a list is now an array of pointers to Tcl_Obj structures. Tcl programmers, for the most part, have not taken advantage of this evolution. Code that uses hash tables where linear arrays would be a more appropriate structure is still extremely common. Moreover, it is difficult to update lists in place, even if their internal representations are known not to be shared. One example of this difficulty is seen in the discussions [] of how best to shuffle a list of items. The discussion began with a naïve implementation of Jon Bentley's method of performing random swaps: ICBwcm9jIHNodWZmbGUxIHsgbGlzdCB9IHs=ICAgICAgc2V0IG4gW2xsZW5ndGggJGxpc3RdICAgICAgZm9yIHsgc2V0IGkgMCB9IHsgJGkgPCAkbiB9IHsgaW5jciBpIH0gew==ICAgICAgICAgIHNldCBqIFtleHByIHtpbnQocmFuZCgpKiRuKX1dICAgICAgICAgIHNldCB0ZW1wIFtsaW5kZXggJGxpc3QgJGpdICAgICAgICAgIHNldCBsaXN0IFtscmVwbGFjZSAkbGlzdCAkaiAkaiBbbGluZGV4ICRsaXN0ICRpXV0=ICAgICAgICAgIHNldCBsaXN0IFtscmVwbGFjZSAkbGlzdCAkaSAkaSAkdGVtcF0=ICAgICAgfQ==ICAgICAgcmV0dXJuICRsaXN0ICB9 Aside from the fact that the syntax obscures what the program is doing, the implementation suffers from an obscure performance problem. When the lreplace calls in the shuffle1 procedure are executed, the internal representation of list has two references: the value of the variable, and the parameter passed to lreplace. The multiple references force lreplace to copy the list, leading to quadratic performance when large lists are shuffled. It is possible, albeit difficult, to alleviate this problem by careful management of the lifetime of Tcl_Obj structures, but this change complicates the code. The simplest way to fix the performance is probably to use Donal Fellows's implementation of the K combinator: IHByb2MgSyB7IHggeSB9IHsgc2V0IHggfQ== which allows the caller of lreplace to extract the value of list, change the value of list so that the extracted value is unshared, and then pass the extracted value as a parameter to lreplace: ICBwcm9jIHNodWZmbGUxYSB7IGxpc3QgfSB7ICAgICAgc2V0IG4gW2xsZW5ndGggJGxpc3RdICAgICAgZm9yIHsgc2V0IGkgMCB9IHsgJGkgPCAkbiB9IHsgaW5jciBpIH0gew==ICAgICAgICAgIHNldCBqIFtleHByIHtpbnQocmFuZCgpKiRuKX1dICAgICAgICAgIHNldCB0ZW1wMSBbbGluZGV4ICRsaXN0ICRqXQ==ICAgICAgICAgIHNldCB0ZW1wMiBbbGluZGV4ICRsaXN0ICRpXQ==ICAgICAgICAgIHNldCBsaXN0IFtscmVwbGFjZSBbSyAkbGlzdCBbc2V0IGxpc3Qge31dXSAkaiAkaiAkdGVtcDJdICAgICAgICAgIHNldCBsaXN0IFtscmVwbGFjZSBbSyAkbGlzdCBbc2V0IGxpc3Qge31dXSAkaSAkaSAkdGVtcDFdICAgICAgfQ==ICAgICAgcmV0dXJuICRsaXN0ICB9 Now the performance of the code is O(n) where n is the length of the list, but the programmer's intent has been seriously obscured! Moreover, the performance is still rather poor: Tcl makes an atrocious showing, for instance, in Doug Bagley's 'Great Computer Language Shootout' [].

This TIP proposes an 'lset' command with the syntax: ICBsc2V0IHZhck5hbWUgaW5kZXhMaXN0IHZhbHVl or ICBsc2V0IHZhck5hbWUgaW5kZXgxIGluZGV4Mi4uLiB2YWx1ZQ== where: varName is the name of a variable in the caller's scope. If objc==4, then the indexList parameter is interpreted as a list of index arguments; if objc>4, then the index arguments are inline on the command line. In either case, Each index argument is an index in the content of varName or one of its sublists (see below). The format of index is either an integer whose value is at least zero and less than the length of the corresponding list, or else the literal string end, optionally followed with a hyphen and an integer whose value is at least zero and less than the length of the corresponding list. value is a value that is to be stored as a list element. The return value of the command, if successful, is the new value of varName. The simplest form of the command: ICBsc2V0IHZhck5hbWUgaW5kZXggdmFsdWU= replaces, in place, the index element of varName with the specified value. For example, the code: ICBzZXQgeCB7YSBiIGN9ICBsc2V0IHggMSBk results in x having the value a d c. The result, except for performance considerations and the details of error reporting, is roughly the same as the Tcl code: ICBwcm9jIGxzZXQgeyB2YXJOYW1lIGluZGV4IHZhbHVlIH0gew==ICAgICAgdXB2YXIgMSAkdmFyTmFtZSBsaXN0ICAgICAgc2V0IGxpc3QgW2xyZXBsYWNlICRsaXN0ICRpbmRleCAkaW5kZXggJHZhbHVlXQ==ICAgICAgcmV0dXJuICRsaXN0ICB9 except that where the lreplace command permits indices outside the existing list elements, the proposed lset command forbids them. If multiple index arguments are supplied to the lset command, they refer to successive sublists in a hierarchical fashion. Thus, ICAgbHNldCB2YXJOYW1lICRpICRqIHZhbHVl or, equivalently, ICAgbHNldCB2YXJOYW1lIFtsaXN0ICRpICRqXSB2YWx1ZQ== asks to change the value of the jth element in the ith sublist of varName. Hence, the code: ICAgc2V0IHgge3thIGIgY30ge2QgZSBmfSB7ZyBoIGl9fQ==ICAgbHNldCB4IDEgMSBqOyAjIC1vci0gbHNldCB4IHsxIDF9IGo= changes the value of x to ICAge2EgYiBjfSB7ZCBqIGZ9IHtnIGggaX0= and the code ICAgc2V0IHkge3t7YSBifSB7YyBkfX0ge3tlIGZ9IHtnIGh9fX0=ICAgbHNldCB5IDEgMCAxIGk7ICMgLW9yLSBsc2V0IHkgezEgMCAxfSBp changes the value of y to ICAge3thIGJ9IHtjIGR9fSB7e2UgaX0ge2cgaH19 This notation also dovetails prettily with the extension of the lindex command proposed in . The command ICAgbGluZGV4ICR5IDEgMCAxOyAjIC1vci0gbGluZGV4IHkgezEgMCAxfQ== will extract the element that is set by the command ICAgbHNldCAkeSAxIDAgMSAkdmFsdWU= The lset command will throw an error and leave the variable unchanged if it is presented with fewer than three arguments, if any of the index arguments is out of range or ill-formed, or if any of the data being manipulated cannot be converted to lists. It will throw an error after modifying the variable if any write trace on the variable fails. With the proposed lset command, the procedure to shuffle a list becomes much more straightforward: ICBwcm9jIHNodWZmbGUxYiB7IGxpc3QgfSB7ICAgICAgc2V0IG4gW2xsZW5ndGggJGxpc3RdICAgICAgZm9yIHsgc2V0IGkgMCB9IHsgJGkgPCAkbiB9IHsgaW5jciBpIH0gew==ICAgICAgICAgIHNldCBqIFtleHByIHtpbnQocmFuZCgpKiRuKX1dICAgICAgICAgIHNldCB0ZW1wIFtsaW5kZXggJGxpc3QgJGpdICAgICAgICAgIGxzZXQgbGlzdCAkaiBbbGluZGV4ICRsaXN0ICRpXQ==ICAgICAgICAgIGxzZXQgbGlzdCAkaSAkdGVtcA==ICAgICAgfQ==ICAgICAgcmV0dXJuICRsaXN0ICB9 The given implementation copies the list only once, the first time that the line: ICAgICAgICAgIGxzZXQgbGlzdCAkaiBbbGluZGV4ICRsaXN0ICRpXQ== is executed. Thereafter, the list is an unshared object, and the replacements are performed in place.

The author has implemented a simpler variant of the proposed command as an object command, and also proposes to bytecode compile it, although the implementation of bytecode compilation is incomplete. The reference implementation also does not yet expand objv[2] as a list in the case where objc==4, and is known to have memory leaks where ill-formed index arguments are presented. It is given here as concept code and to present its impact on performance of some common list operations. (Obviously, it will be completed and reviewed with the relevant maintainers prior to being committed to the Core.) The core of the implementation is the following procedure: aW50VGNsX0xzZXRPYmpDbWQoIGNsaWVudERhdGEsIGludGVycCwgb2JqYywgb2JqdiApICAgIENsaWVudERhdGEgY2xpZW50RGF0YTsgICAgICAvKiBOb3QgdXNlZC4gKi8=ICAgIFRjbF9JbnRlcnAgKmludGVycDsgICAgICAgICAvKiBDdXJyZW50IGludGVycHJldGVyLiAqLw==ICAgIGludCBvYmpjOyAgICAgICAgICAgICAgICAgICAvKiBOdW1iZXIgb2YgYXJndW1lbnRzLiAqLw==ICAgIFRjbF9PYmogKkNPTlNUIG9ianZbXTsgICAgICAvKiBBcmd1bWVudCB2YWx1ZXMuICovew==ICAgIFRjbF9PYmoqIGxpc3RQdHI7ICAgICAgICAgICAvKiBQb2ludGVyIHRvIHRoZSBsaXN0IGJlaW5nIGFsdGVyZWQuICovICAgIFRjbF9PYmoqIHN1Ykxpc3RQdHI7ICAgICAgICAvKiBQb2ludGVyIHRvIGEgc3VibGlzdCBvZiB0aGUgbGlzdCAqLw==ICAgIFRjbF9PYmoqIGZpbmFsVmFsdWVQdHI7ICAgICAvKiBWYWx1ZSBmaW5hbGx5IGFzc2lnbmVkIHRvIHRoZSB2YXJpYWJsZSAqLw==ICAgIGludCBpbmRleDsgICAgICAgICAgICAgICAgICAvKiBJbmRleCBvZiB0aGUgZWxlbWVudCBiZWluZyByZXBsYWNlZCAqLw==ICAgIGludCByZXN1bHQ7ICAgICAgICAgICAgICAgICAvKiBSZXN1bHQgdG8gcmV0dXJuIGZyb20gdGhpcyBmdW5jdGlvbiAqLw==ICAgIGludCBsaXN0TGVuOyAgICAgICAgICAgICAgICAvKiBMZW5ndGggb2YgYSBsaXN0IGJlaW5nIGV4YW1pbmVkICovICAgIFRjbF9PYmoqKiBlbGVtUHRyczsgICAgICAgICAvKiBQb2ludGVycyB0byB0aGUgZWxlbWVudHMgb2YgYQ==ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKiBsaXN0IGJlaW5nIGV4YW1pbmVkICovICAgIGludCBpOw==ICAgIC8qIENoZWNrIHBhcmFtZXRlciBjb3VudCAqLw==ICAgIGlmICggb2JqYyA8IDQgKSB7ICAgICAgICBUY2xfV3JvbmdOdW1BcmdzKCBpbnRlcnAsIDEsIG9ianYsICJsaXN0VmFyIGluZGV4ID9pbmRleC4uLj8gdmFsdWUiICk7ICAgICAgICByZXR1cm4gVENMX0VSUk9SOw==ICAgIH0=ICAgIC8qIExvb2sgdXAgdGhlIGxpc3QgdmFyaWFibGUgKi8=ICAgIGxpc3RQdHIgPSBUY2xfT2JqR2V0VmFyMiggaW50ZXJwLCBvYmp2WyAxIF0sIChUY2xfT2JqKikgTlVMTCw=ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgVENMX0xFQVZFX0VSUl9NU0cgKTs=ICAgIGlmICggbGlzdFB0ciA9PSBOVUxMICkgew==ICAgICAgICByZXR1cm4gVENMX0VSUk9SOw==ICAgIH0=ICAgIC8qIE1ha2Ugc3VyZSB0aGF0IHRoZSBsaXN0IHZhbHVlIGlzIHVuc2hhcmVkLiAqLw==ICAgIGlmICggVGNsX0lzU2hhcmVkKCBsaXN0UHRyICkgKSB7ICAgICAgICBsaXN0UHRyID0gVGNsX0R1cGxpY2F0ZU9iaiggbGlzdFB0ciApOw==ICAgIH0=ICAgIGZpbmFsVmFsdWVQdHIgPSBsaXN0UHRyOw==ICAgIC8qICAgICAqIElmIHRoZXJlIGFyZSBtdWx0aXBsZSAnaW5kZXgnIGFyZ3MsIGhhbmRsZSBlYWNoIGFyZyBleGNlcHQgdGhlICAgICAqIGxhc3QgYnkgZGl2aW5nIGludG8gYSBzdWJsaXN0Lg==ICAgICAqLw==ICAgIGZvciAoIGkgPSAyOyA7ICsraSApIHs=ICAgICAgICAvKiBUYWtlIGFwYXJ0IHRoZSBsaXN0ICovICAgICAgICByZXN1bHQgPSBUY2xfTGlzdE9iakdldEVsZW1lbnRzKCBpbnRlcnAsIGxpc3RQdHIsICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAmbGlzdExlbiwgJmVsZW1QdHJzICk7ICAgICAgICBpZiAoIHJlc3VsdCAhPSBUQ0xfT0sgKSB7ICAgICAgICAgICAgcmV0dXJuIHJlc3VsdDs=ICAgICAgICB9ICAgICAgICAvKiBEZXJpdmUgdGhlIGluZGV4IG9mIHRoZSByZXF1ZXN0ZWQgc3VibGlzdCAqLw==ICAgICAgICByZXN1bHQgPSBUY2xHZXRJbnRGb3JJbmRleCggaW50ZXJwLCBvYmp2W2ldLCAobGlzdExlbiAtIDEpLCAmaW5kZXggKTs=ICAgICAgICBpZiAoIHJlc3VsdCAhPSBUQ0xfT0sgKSB7ICAgICAgICAgICAgcmV0dXJuIHJlc3VsdDs=ICAgICAgICB9ICAgICAgICBpZiAoICggaW5kZXggPCAwICkgfHwgKCBpbmRleCA+PSBsaXN0TGVuICkgKSB7ICAgICAgICAgICAgVGNsX1NldE9ialJlc3VsdCggaW50ZXJwLA==ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgVGNsX05ld1N0cmluZ09iaiggImxpc3QgaW5kZXggb3V0IG9mIHJhbmdlIiw=ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLTEgKSApOw==ICAgICAgICAgICAgcmV0dXJuIFRDTF9FUlJPUjs=ICAgICAgICB9ICAgICAgICAvKiBCcmVhayBvdXQgb2YgdGhlIGxvb3AgaWYgd2UndmUgZXh0cmFjdGVkIHRoZSBpbm5lcm1vc3Qgc3VibGlzdC4gKi8=ICAgICAgICBpZiAoIGkgPj0gKCBvYmpjIC0gMiApICkgew==ICAgICAgICAgICAgYnJlYWs7ICAgICAgICB9ICAgICAgICAvKg==ICAgICAgICAgKiBFeHRyYWN0IHRoZSBhcHByb3ByaWF0ZSBzdWJsaXN0LCBhbmQgbWFrZSBzdXJlIHRoYXQgaXQgaXMgdW5zaGFyZWQuICAgICAgICAgKi8=ICAgICAgICBzdWJMaXN0UHRyID0gZWxlbVB0cnNbIGluZGV4IF07ICAgICAgICBpZiAoIFRjbF9Jc1NoYXJlZCggc3ViTGlzdFB0ciApICkgew==ICAgICAgICAgICAgc3ViTGlzdFB0ciA9IFRjbF9EdXBsaWNhdGVPYmooIHN1Ykxpc3RQdHIgKTs=ICAgICAgICAgICAgcmVzdWx0ID0gVGNsX0xpc3RPYmpTZXRFbGVtZW50KCBpbnRlcnAsIGxpc3RQdHIsIGluZGV4LA==ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzdWJMaXN0UHRyICk7ICAgICAgICAgICAgaWYgKCByZXN1bHQgIT0gVENMX09LICkgew==ICAgICAgICAgICAgICAgIHJldHVybiBUQ0xfRVJST1I7ICAgICAgICAgICAgfQ==ICAgICAgICB9IGVsc2Ugew==ICAgICAgICAgICAgVGNsX0ludmFsaWRhdGVTdHJpbmdSZXAoIGxpc3RQdHIgKTs=ICAgICAgICB9ICAgICAgICBsaXN0UHRyID0gc3ViTGlzdFB0cjs=ICAgIH0=ICAgIC8qIFN0b3JlIHRoZSByZXN1bHQgaW4gdGhlIGxpc3QgZWxlbWVudCAqLw==ICAgIHJlc3VsdCA9IFRjbF9MaXN0T2JqU2V0RWxlbWVudCggaW50ZXJwLCBsaXN0UHRyLCBpbmRleCwgb2JqdltvYmpjLTFdICk7ICAgIGlmICggcmVzdWx0ICE9IFRDTF9PSyApIHs=ICAgICAgICByZXR1cm4gcmVzdWx0Ow==ICAgIH0=ICAgIC8qIEZpbmFsbHksIHVwZGF0ZSB0aGUgdmFyaWFibGUgc28gdGhhdCB0cmFjZXMgZmlyZS4gKi8=ICAgIGxpc3RQdHIgPSBUY2xfT2JqU2V0VmFyMiggaW50ZXJwLCBvYmp2WzFdLCBOVUxMLCBmaW5hbFZhbHVlUHRyLA==ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgVENMX0xFQVZFX0VSUl9NU0cgKTs=ICAgIGlmICggbGlzdFB0ciA9PSBOVUxMICkgew==ICAgICAgICByZXR1cm4gVENMX0VSUk9SOw==ICAgIH0=ICAgICAgIA==ICAgIFRjbF9TZXRPYmpSZXN1bHQoIGludGVycCwgbGlzdFB0ciApOw==ICAgIHJldHVybiByZXN1bHQ7fQ== The procedure depends on a new service function, Tcl_ListObjSetElement: aW50VGNsX0xpc3RPYmpTZXRFbGVtZW50KCBpbnRlcnAsIGxpc3RQdHIsIGluZGV4LCB2YWx1ZVB0ciApICAgIFRjbF9JbnRlcnAqIGludGVycDsgICAgICAgICAvKiBUY2wgaW50ZXJwcmV0ZXI7IHVzZWQgZm9yIGVycm9yIHJlcG9ydGluZw==ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKiBpZiBub3QgTlVMTCAqLw==ICAgIFRjbF9PYmoqIGxpc3RQdHI7ICAgICAgICAgICAvKiBMaXN0IG9iamVjdCBpbiB3aGljaCBlbGVtZW50IHNob3VsZCBiZQ==ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKiBzdG9yZWQgKi8=ICAgIGludCBpbmRleDsgICAgICAgICAgICAgICAgICAvKiBJbmRleCBvZiBlbGVtZW50IHRvIHN0b3JlICovICAgIFRjbF9PYmoqIHZhbHVlUHRyOyAgICAgICAgICAvKiBUY2wgb2JqZWN0IHRvIHN0b3JlIGluIHRoZSBkZXNpZ25hdGVkICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKiBsaXN0IGVsZW1lbnQgKi8=ew==ICAgIGludCByZXN1bHQ7ICAgICAgICAgICAgICAgICAvKiBSZXR1cm4gdmFsdWUgZnJvbSB0aGlzIGZ1bmN0aW9uICovICAgIExpc3QqIGxpc3RSZXBQdHI7ICAgICAgICAgICAvKiBJbnRlcm5hbCByZXByZXNlbnRhdGlvbiBvZiB0aGUgbGlzdA==ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKiBiZWluZyBtb2RpZmllZCAqLw==ICAgIFRjbF9PYmoqKiBlbGVtUHRyczsgICAgICAgICAvKiBQb2ludGVycyB0byBlbGVtZW50cyBvZiB0aGUgbGlzdCAqLw==ICAgIGludCBlbGVtQ291bnQ7ICAgICAgICAgICAgICAvKiBOdW1iZXIgb2YgZWxlbWVudHMgaW4gdGhlIGxpc3QgKi8=ICAgIC8qIEVuc3VyZSB0aGF0IHRoZSBsaXN0UHRyIHBhcmFtZXRlciBkZXNpZ25hdGVzIGFuIHVuc2hhcmVkIGxpc3QgKi8=ICAgIGlmICggVGNsX0lzU2hhcmVkKCBsaXN0UHRyICkgKSB7ICAgICAgICBwYW5pYyggIlRjbF9MaXN0T2JqU2V0RWxlbWVudCBjYWxsZWQgd2l0aCBzaGFyZWQgb2JqZWN0IiApOw==ICAgIH0=ICAgIGlmICggbGlzdFB0ci0+dHlwZVB0ciAhPSAmdGNsTGlzdFR5cGUgKSB7ICAgICAgICByZXN1bHQgPSBTZXRMaXN0RnJvbUFueSggaW50ZXJwLCBsaXN0UHRyICk7ICAgICAgICBpZiAoIHJlc3VsdCAhPSBUQ0xfT0sgKSB7ICAgICAgICAgICAgcmV0dXJuIHJlc3VsdDs=ICAgICAgICB9ICAgIH0=ICAgIGxpc3RSZXBQdHIgPSAoTGlzdCopIGxpc3RQdHItPmludGVybmFsUmVwLm90aGVyVmFsdWVQdHI7ICAgIGVsZW1QdHJzID0gbGlzdFJlcFB0ci0+ZWxlbWVudHM7ICAgIGVsZW1Db3VudCA9IGxpc3RSZXBQdHItPmVsZW1Db3VudDs=ICAgIC8qIEVuc3VyZSB0aGF0IHRoZSBpbmRleCBpcyBpbiBib3VuZHMgKi8=ICAgIGlmICggaW5kZXggPCAwIHx8IGluZGV4ID49IGVsZW1Db3VudCApIHs=ICAgICAgICBpZiAoIGludGVycCAhPSBOVUxMICkgew==ICAgICAgICAgICAgVGNsX1NldE9ialJlc3VsdCggaW50ZXJwLA==ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgVGNsX05ld1N0cmluZ09iaiggImxpc3QgaW5kZXggb3V0IG9mIHJhbmdlIiw=ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLTEgKSApOw==ICAgICAgICAgICAgcmV0dXJuIFRDTF9FUlJPUjs=ICAgICAgICB9ICAgIH0=ICAgIC8qIEFkZCBhIHJlZmVyZW5jZSB0byB0aGUgbmV3IGxpc3QgZWxlbWVudCAqLw==ICAgIFRjbF9JbmNyUmVmQ291bnQoIHZhbHVlUHRyICk7ICAgIC8qIFJlbW92ZSBhIHJlZmVyZW5jZSBmcm9tIHRoZSBvbGQgbGlzdCBlbGVtZW50ICovICAgIFRjbF9EZWNyUmVmQ291bnQoIGVsZW1QdHJzWyBpbmRleCBdICk7ICAgIC8qIFN0YXNoIHRoZSBuZXcgb2JqZWN0IGluIHRoZSBsaXN0ICovICAgIGVsZW1QdHJzWyBpbmRleCBdID0gdmFsdWVQdHI7ICAgIC8qIEludmFsaWRhdGUgYW5kIGZyZWUgYW55IG9sZCBzdHJpbmcgcmVwcmVzZW50YXRpb24gKi8=ICAgIFRjbF9JbnZhbGlkYXRlU3RyaW5nUmVwKCBsaXN0UHRyICk7ICAgIHJldHVybiBUQ0xfT0s7ICAgIA==fQ== Even without bytecode compilation, the performance improvement of array-based applications that can be achieved by the lset command is substantial. The following table shows run times in microseconds (on a 550 MHz Pentium III laptop, running a modified Tcl 8.4 on Windows NT 4.0, Service Pack #6) of the three implementations of shuffle that appear in this TIP. ICAgICAgICAgICAgICAgICAgICBSVU4gVElNRVMgSU4gTUlDUk9TRUNPTkRTICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBWZXJzaW9uICAgICAgICAgICAgICAgICAgICAgc2h1ZmZsZTEgICAgICAgc2h1ZmZsZTFhICAgICAgIHNodWZmbGUxYg==ICAgICAgICAgICAgICAgICAgICAgKE5haXZlKSAgICAgIChLIGNvbWJpbmF0b3IpICAobHNldCBjb21tYW5kKQ==IExpc3QgbGVuZ3RoICAgICAgICAxICAgICAgICAgICAgICAgICAyNiAgICAgICAgICAgICAgIDMyICAgICAgICAgICAgICAyNyAgICAgICAgICAgIA==ICAgICAgIDEwICAgICAgICAgICAgICAgIDEwOCAgICAgICAgICAgICAgMTUyICAgICAgICAgICAgIDEwMQ==ICAgICAgMTAwICAgICAgICAgICAgICAgMTYyNyAgICAgICAgICAgICAxNDYyICAgICAgICAgICAgIDkzNg==ICAgICAxMDAwICAgICAgICAgICAgIDExNzgzMSAgICAgICAgICAgIDE0Nzg5ICAgICAgICAgICAgOTU3NA==ICAgIDEwMDAwICAgICAgIFRlc3Qgc3RvcHBlZCAgICAgICAgICAgMTUyODUzICAgICAgICAgICA5NjkxMg== Similar (30-50%) improvements are observed on many of the array related benchmarks that have been proposed. Bytecode compilation is expected to produce even greater improvements. Another area where lset can achieve a major performance gain is in memory usage. The author of this TIP has benchmarked competing implementations of heapsort, one using Tcl arrays, and the other using lset to manipulate lists as linear arrays. When sorting 80000 elements, the Tcl-array-based implementation used 12.7 megabytes of memory; the list-based implementation was faster and used only 5.6 megabytes. The explanation is simple: each entry in the hash table requires an allocated block of twenty bytes of memory, plus the space required for the hash key. The hash key is a string, and requires at least six bytes. When both of these objects are aligned and padded with the overheads imposed by ckalloc, they require about 80 bytes of memory on the Windows NT platform. The memory cost of an element of a Tcl list, by comparison, is four bytes to hold the pointer to the object.

There are several objections that can be foreseen to this proposal. Why implement the command in the Core and not as an extension?In a word, performance. At the present state of Tcl development, only Core commands can be bytecoded. The cost of the hash table lookups in the Tcl_ObjGetVar2 and Tcl_ObjSetVar2 calls is significant, and can be eliminated from many common usages by the bytecode compiler. Since this command is likely to appear in inner loops, it is important to squeeze every bit of possible performance out of it.Why a new command in the global namespace?The author of this TIP feels that having a single added command that is parallel to the existing list commands is not polluting the namespace excessively. It would be a shame if this proposal founders upon the Naming of Names.Why a new command, rather than including this functionality in the proposed functionality of an extensible command for list manipulation?The author of this TIP has yet to see a formal proposal of any extensible list manipulation command; the closest thing appears to be Andreas Kupries's listx package []. Given the size and complexity of any such modification, it is unlikely that it will be available in the Core in time for an 8.4 release. The performance improvements achievable by the lset command are needed urgently.Isn't this warmed over?Several objectors to indicated that they were willing to consider list element assignment implemented as a new command.Doesn't this proposal depend on multiple index arguments to lindex'' ()?This proposal can stand alone. If multiple index arguments to lindex are also accepted, the resulting symmetry is pleasing. Having multiple index args to lset is much more important, because it is horribly difficult to implement equivalent functionality in pure Tcl without introducing excessive calls to Tcl_DuplicateObj. In fact, the reference implementation of lset presented in this TIP was motivated by the fact that its author gave up on the task and resorted to C.

Having two versions of the syntax for the lset command is perhaps unattractive, but neither can be left out effectively. The syntax where the indices are packaged as a single list allows a cursor into complex list structure to be maintained in a single variable. The list element that the cursor designates can be altered with a single call to the lset command, without needing to resort to eval (a command that is both expensive and dangerous) to expand the indices inline. The syntax where each index is a first-class object is motivated by the performance of array-based algorithms. Programmers who are using lists as arrays know exactly how many subscripts they have, and in fact are generally iterating through them. A typical sort of usage might be the naïve matrix multiplication shown below. ICMgQ29uc3RydWN0IGEgbWF0cml4IHdpdGggJ3Jvd3MnIHJvd3MgYW5kICdjb2x1bW5zJyBjb2x1bW5zICMgaGF2aW5nIGFuIGluaXRpYWwgdmFsdWUgb2YgJ2luaXRDZWxsVmFsdWUnIGluIGVhY2ggY2VsbC4=IA==IHByb2MgbWF0cml4IHsgcm93cyBjb2x1bW5zIHsgaW5pdENlbGxWYWx1ZSB7fSB9IH0gew==ICAgICBzZXQgb25lUm93IHt9ICAgICBmb3IgeyBzZXQgaSAwIH0geyAkaSA8ICRjb2x1bW5zIH0geyBpbmNyIGkgfSB7ICAgICAgICAgbGFwcGVuZCBvbmVSb3cgJGluaXRDZWxsVmFsdWU=ICAgICB9ICAgICBzZXQgbWF0cml4IHt9ICAgICBmb3IgeyBzZXQgaSAwIH0geyAkaSA8ICRyb3dzIH0geyBpbmNyIGkgfSB7ICAgICAgICAgbGFwcGVuZCBtYXRyaXggJG9uZVJvdw==ICAgICB9ICAgICByZXR1cm4gJG1hdHJpeA==IH0=IA==ICMgTXVsdGlwbHkgdHdvIG1hdHJpY2VzIA==IHByb2MgbWF0bXVsdCB7IHggeSB9IHs=IA==ICAgICBzZXQgbSBbbGxlbmd0aCAkeF07ICAgICAgICAgICAgICAgICAjIE51bWJlciBvZiByb3dzIG9mIGxlZnQgbWF0cml4ICAgICBzZXQgbiBbbGxlbmd0aCBbbGluZGV4ICR4IDBdXTsgICAgICAjIE51bWJlciBvZiBjb2x1bW5zIG9mIGxlZnQgbWF0cml4IA==ICAgICBpZiB7ICRuICE9IFtsbGVuZ3RoICR5XSB9IHs=ICAgICAgICAgcmV0dXJuIC1jb2RlIGVycm9yICJyYW5rIGVycm9yOiBsZWZ0IG9wZXJhbmQgaGFzICRuIGNvbHVtbnNcICAgICAgICAgICAgICAgICAgICAgICAgICAgICB3aGlsZSByaWdodCBvcGVyYW5kIGhhcyBbbGxlbmd0aCAkeV0gcm93cyI=ICAgICB9IA==ICAgICBzZXQgayBbbGxlbmd0aCBbbGluZGV4ICR5IDBdXTsgICAgICAjIE51bWJlciBvZiBjb2x1bW5zIG9mIHJpZ2h0IG1hdHJpeA==IA==ICAgICAjIENvbnN0cnVjdCBhIG1hdHJpeCB0byBob2xkIHRoZSBwcm9kdWN0IA==ICAgICBzZXQgcHJvZHVjdCBbbWF0cml4ICRtICRrXQ==IA==ICAgICBmb3IgeyBzZXQgaSAwIH0geyAkaSA8ICRtIH0geyBpbmNyIGkgfSB7ICAgICAgICAgZm9yIHsgc2V0IGogMCB9IHsgJGogPCAkayB9IHsgaW5jciBqIH0gew==ICAgICAgICAgICAgIGxzZXQgcHJvZHVjdCAkaSAkaiAwLjA=ICAgICAgICAgICAgIGZvciB7IHNldCByIDAgfSB7ICRyIDwgJG4gfSB7IGluY3IgciB9IHs=ICAgICAgICAgICAgICAgICBzZXQgdGVybSBbZXhwciB7IFtsaW5kZXggJHggJGkgJHJdICogW2xpbmRleCAkeSAkciAkal0gfV0=ICAgICAgICAgICAgICAgICBsc2V0IHByb2R1Y3QgJGkgJGogW2V4cHIgeyBbbGluZGV4ICRwcm9kdWN0ICRpICRqXSArICR0ZXJtIH1dICAgICAgICAgICAgIH0=ICAgICAgICAgfQ==ICAgICB9IA==ICAgICByZXR1cm4gJHByb2R1Y3Q=IH0= Note how we have an [lset] operation in the innermost loop, executed (m*n*k) times. If in this instance, we have to write: ICAgICAgICAgICAgICAgICBzZXQgaW5kaWNlcyBbbGlzdCAkaSAkal0=ICAgICAgICAgICAgICAgICBsc2V0IHByb2R1Y3QgJGluZGljZXMgXA==ICAgICAgICAgICAgICAgICAgICAgW2V4cHIgeyBbbGluZGV4ICRwcm9kdWN0ICRpbmRpY2VzXSArICR0ZXJtIH1d in place of the [lset] shown above, we add the cost of forming the list of indices to the cost of the inner loop. This cost is not to be sneezed at -- it's two expensive calls to ckalloc. (The cost can be avoided, at some cost in readability, by maintaning a variable containing the index list, and altering its elements with other uses of [lset].) Richard Suchenwirth suggested the compromise that appears in this proposal. This scheme will perilous to performance if implemented naively. If the implementation of [lset] simply calls Tcl_ListObjGetElements, look what happens to the inner loop of our shuffle1b procedure: ICAgICAgZm9yIHsgc2V0IGkgMCB9IHsgJGkgPCAkbiB9IHsgaW5jciBpIH0gew==ICAgICAgICAgIHNldCBqIFtleHByIHtpbnQocmFuZCgpKiRuKX1dICAgICAgICAgIHNldCB0ZW1wIFtsaW5kZXggJGxpc3QgJGpdICAgICAgICAgIGxzZXQgbGlzdCAkaiBbbGluZGV4ICRsaXN0ICRpXQ==ICAgICAgICAgIGxzZXQgbGlzdCAkaSAkdGVtcA==ICAgICAgfQ== Initially, {set i 0} sets i to the constant "0"; it is a string.Evaluating the conditional {$i < $n} will shimmer i to an integer; now it's an integer. (We had to do a call to strtol here.)The [lindex $list $i] call now has to consider $i as a list of indices, and shimmers it to the list. This discards the internal rep, parses the string rep into a list, and then reconverts its first element to an integer.OK, now the 'lset' is happy, and no further shimmering occurs...... until we get to the {incr i}. Now we go back to the string rep once again, shimmer it to an integer (yet another call to strtol), and invalidate the string rep because we've incremented the integer.Now we get back into the [lindex] once again, and need a list rep. This time, we have to format the integer as a string, parse it as a list, take the object representing element 0, and reparse that as an integer. This sequence has converted the integer to and from a string, and performed four calls to ckalloc, but resulted in the same integer that we started with! It is possible for a sufficiently smart compromise implementation to avoid all this shimmering. In the case where objc==4, the lset command must: Test whether objv[2] designates an object whose internal representation holds an integer. If so, simply use it as an index.Test whether objv[2] designates an object whose internal representation holds a list. If so, perform the recursive extraction of indexed elements from sublists described above.Form the string representation of objv[2] and test whether it is end or end- followed by an integer. If so, use it as an index.Attempt to coerce objv[2] to an integer; if successful, use the result as an integer.Attempt to coerce objv[2] to a list; if successful, use the result as an index list.Report a malformed index argument; the indexList parameter is not a well-formed list. This logic handles all the cases of singleton lists transparently; it is effectively a simple-minded type inference that optimizes away needless conversions. With it in place, none of the lset examples shown in this TIP will suffer from type shimmering. In the event that the related is approved, the logic for parsing an index list will likely be combined with that used in the lindex command. Bytecoding variadic commands like lset presents some interesting technical challenges; a discussion in progress on the Tcl'ers Wiki [] is recording the design decisions being made for bytecoding lset so that they can be applied to similar commands in the future.

, .

This TIP has undergone several revisions by the original author. The most significant was made on 20 May 2001, where the syntax was revised to allow for either several indices inline on the command line or a list of indices.

This document has been placed in the public domain.