Anyway, check this out. I have been tasked with writing a SQL Server database module for Drupal 6. This is not fun. It’s slow, Drupal is written poorly for abstraction, and there are odd idiosyncracies one has to deal with. But that’s not what I want to tell you about.

This is the vargrind output between a semi-working mssql installation and a working mysql installation. That 400 number is the number of queries the MSSQL module needs to do to render the ‘My Account’ page. For comparison, MySQL only requires about 40.

(The reason it’s so high is because one must select the error code to check for an error, then select the code meaning from the database if there was an actual error. For every query.)

Also, this is kind of long.)

I have the following association set that represents a document repository.


User 1-n Folders
User 1-n Roles
Folder 1-n Documents
Documents n-n Brands


Folders n-n Viewer Roles
Folders n-n Manager Roles


Lastly, Users n -(polymorphic through)- n Brands

In order to display a tree of Folders a given user can view, that User must
be a manager or viewer of a Folder, and his Brands contain Brand of at
least one Document under that Folder (a Document associated with a Brand under a folder lets the user view that Folder iff that user has the Brand.)

So the way this was originally implemented was a recursive loop
something like this:

def show_subfolders(folder)
User.folders.each do |folder|
show_subfolders(folder)
end
end

Class User
def folders
Folder.find(:all,:conditions => ‘parent = folder’,
:include => [:manager_roles, :viewer_roles, {:documents => :brands]]
).each do
|folder|
if method_to_check_brands(folder) && method_to_check_view_perms(folder)
yield(folder)
end
end
end

The eventual problems that developed with this logic were the following:

1. second-level includes are problematic in Rails, this generated
   outer joins against seven tables
2. massive join, recursively called. I believe the Big O* is (c!)^n,
   glancing quickly
3. Indexes don’t help, because the outer joins sort of blow the
   performance gains away
4. All data is being selected, so just sheer amount of data transfer
   slowed things down
5. The yield adds a lot of crap to the callstack
6. Brand and View/Manage permissioning was already happening in the code

With 245 documents under 30 folders, mysql logged slow queries that
had to examine 320,000 rows — and that was multiple times.

The solution? Remove the :includes. Since permissioning and
restriction was already happening in code, I let Rails do its own
thing and select when necessary. This lowered the response time from
10+ seconds for page view to 0.5 seconds. (And for what it’s worth,
doing the join statements manually, or removing the second-level :includes only brings it down to 2 or 3 seconds.)

By reducing the stuff that needs to join, indices are are used more
efficiently, and letting Rails do a find for Folder.documents on each
iteration ends up being a lot faster than trying to join with the
view/manage/brands permissions.

The More You Know!

* It’s been over a decade since I actually figured out the order of an algorithm. I’m sort of guessing, here.

dynamic buffer insertion with PHP

Needed to quickly add a notice to every page on a client’s site where there was a single include, but not single-included layout pages (teachers, *sigh*).

I’ve never messed with output buffer stuff before, so this was just sort of surprising to me. It’s insanely useful.

Only took about a minute to code, and two to deploy.

You will notice that in my screenshot, the final function call to ‘list_cats’ at the end is truncated. This is because there are eighteen parameters. Again with the clicking.

Also, there’s an awesome feature in the WP DB object such that if one requeries the same table on the same page, apparently pagination offset is lost. This means that the main index page listing posts cannot also have a recent posts list — because you either will have recent posts starting with whatever the main list offset is, or you will only ever have as many posts on the index page as the recent posts query returns.

WIN!


map <desiredkey> <targetaction>

So, for example, I split windows vertically and want to navigate via ctrl+pagedown or pageup.

Here’s what I did (include < and > when defining)

map <c-pageup> <c-w>h
map <c-pagedown> <c-w>l


Found via a vim wiki search
[ad#Google Adsense small leaderboard]

I’m posting this because I had a hell of a time finding all the info on it.

Update: This is the significant portion of my xorg.conf.
Update, a few years later: The most important thing I’ve learned about dealing with the radeon

Resources:
Radeon driver xorg settings
XGL Troubleshooting
Compiz wiki ATI notes

My desired setup: two CRTs side by side, each running at 1280×1024, giving me an effective desktop of 2560×1024.

For desktop effects to work, I had to run the ‘radeon’ driver, not the ‘ati’ nor ‘fglrx’ drivers. If one uses fglrx, I guess one can get by running XGL instead of AIGLX, but that locked up my X.

First important note: If fglrx was ever installed and you want to use ati or radeon, you need to fully remove fglrx (sudo apt-get remove –purge fglrx).

Second important note: To get a desktop spanning across two monitors, you want to use the MergedFB option in your xorg.conf. This provides Xinerama-like functionality. There are several options with that mode to provide different resolutions on each monitor with spanning.

Third important note: There is an apparent hardware limitation for these cards on texture sizes. The max texture size for 3D applications (including compiz) of 2048×2048). This means that if you have a desktop wider than 2048, you’ll either

1. have effects for half or 2/3 of the desktop/screen with white on the far right, or
2. a fully white screen with odd tearing on effects

There’s not much way around this at this point. Some people can fix it messing with ‘driconf’.
My solution at this point is to have one screen ‘Above’ instead of ‘RightOf’ in my xorg.conf (the two heights combine to 2048, so they fit into memory).

Anyway, that’s about it. I’ll add my xorg.conf to this post, later. I can’t cause I’m on the laptop right now.


# xorg.conf significant sections for dual head + compiz on radeon 9600 XT
# second screen on top of second screen, vertical orientation.

Section "Module"
Load "bitmap"
Load "ddc"
Load "dri"
Load "dbe"
Load "extmod"
Load "freetype"
Load "glx"
Load "int10"
Load "type1"
Load "vbe"
EndSection

Section "Device"
Identifier "Radeon"
Driver "radeon"
BusID "PCI:1:0:0"
Option "No2048Limit"
Option "MergedFB" "true"
Option "MergedXineramaCRT2IsScreen0" "true"
Option "CRT2Position" "Above"
Option "MetaModes" "1280x1024-1280x1024 1024x768-1024x768"

Option "DDCMode" "true"
Option "OpenGLOverlay" "Off"
Option "VideoOverlay" "on"
Option "GLOverlay" "false"
Option "OverlayOnCRTC1" "on"
Option "OverlayOnCRTC2" "on"
Option "MergedDPI" "75 75"
Option "AGPMode" "8"
Option "DRI" "true"
Option "ColorTiling" "on"
Option "EnablePageFlip" "true"
#DO NOT USE WITH RADEON 9600 Option "AccelMethod" "EXA"
Option "XAANoOffscreenPixmaps"
Option "RenderAccel" "true"
EndSection

Section "Monitor"
Identifier "Dell P991 [0]"
Option "DPMS"
HorizSync 30-107
VertRefresh 48-85
EndSection

Section "Screen"
Identifier "Screen0"
Device "Radeon"
Monitor "Dell P991 [0]"
DefaultDepth 24
SubSection "Display"
Virtual 1280 2048
Depth 24
Modes "1280x1024" "1024x768"
EndSubSection
EndSection

Section "ServerLayout"
Identifier "Default Layout"
Screen "Screen0"
InputDevice "Generic Keyboard"
InputDevice "Configured Mouse"
InputDevice "stylus" "SendCoreEvents"
InputDevice "cursor" "SendCoreEvents"
InputDevice "eraser" "SendCoreEvents"
Option "AIGLX" "true"
EndSection

Section "DRI"
Mode 0666
EndSection

Section "Extensions"
Option "Composite" "On"
EndSection


[ad#Google Adsense-1 Block]

This will generate an HTML version of your source code, complete with color scheme. It’s pretty sweet.

Example here (opens a new window)

This just tickles me.

Situation, selecting the first 3 Announcements for particular User Roles at a particular Organization.

There is an announcement table and a meta table containing org/role join information.

The (somewhat crappy) query is like so:
SELECT TOP 3 b.announcement_title FROM announcements b
WHERE b.anouncement_id IN (
SELECT
  DISTINCT a.announcement_id
FROM
  announcements b, announcement_role_org aro
WHERE
  aro.announcement_id = b.announcement_id
   AND
  aro.role_id = $role_id
   AND
  aro.org_id = $org_id
)


Yes, I know it should have been written as a join.

This is the problem:

• When the outer select says TOP 3, the server chugs forever and doesn’t return a result (at least, doesn’t return a result within 2 minutes, I killed the select after that time limit).
• When the outer select says TOP 5 or TOP 6, it returns results fairly instantly

WTF?
For the record, the inner select for the $role and $org I specified only ever return 6 results.

Oh, lastly… if I say NOT IN (returns 210 results), it doesn’t seem to matter what TOP value I have.

WHY???

A Very Contrived Example:

def string_to_integer
  if block_given?
    yield.to_i
  else
    0
  end
end
a = [1,2,3]

A) a.inject string_to_integer { ’5′ } do |i,sum| i + sum end => 11
B) a.inject string_to_integer { |i,sum| i + sum } => undefined method error
C) a.inject string_to_integer do |i,sum| i + sum end => 6

in A, the first brace is sent to the string_to_integer method
in B, the same happens, but the block was actually meant for inject
in C it works as intended.

Just food for thought.


require 'iconv'
require 'charguess'
# Reads an arbitrary file and attempts to convert to a new file in UTF-8
# Sort of quick and dirty
# You should note that Iconv.conv will barf if you specify
# a source encoding that doesn't match the actual source encoding
def poop_out_utf8(infile, outfile)
  if infile == outfile
    puts 'Source and destination are the same, silly!"
    return
  end
  out_file = File.new outfile, 'w'
  begin
    File.open(infile, 'r') do |file|
      file.each_line{|line|
        out_file.puts Iconv.conv('UTF-8', CharGuess::guess(line),line) unless line.nil?
      }
  end
  rescue Exception => e
    puts e
  ensure
    out_file.close
  end
end


You should have iconv already. CharGuess you’ll need to install yourself, most likely. Grab the clib here, and the ruby binding here.

Assuming you’ve extracted the clib to /home/user/libcharguess/ (and done the ./configure/make/sudo make install bit), install the ruby charguess bindings with

ruby extconf.rb --with-charguess-include=/home/user/libcharguess/cpp/
make
make install


Use with care — it doesn’t differentiate between partial names. So if you want to kill all processes with ‘mini’ and you just specify ‘ini’, you will also attempt to kill ‘init’ (though that shouldn’t work).

(Also: The process group flag for ‘killall’ should do this, but it doesn’t work for Amarok, on my system.)

This isn’t really a comprehensive review, just an exclamation over the techniques involved in the implementation of the software.

Jinzora is a PHP+Apache+(MySQL/Postgres/SQLite/etc) application that indexes and manages your music library, and provides streaming access to said library over a network. It’s a great idea — you can have a music server set up in the house and let satellite computers stream a music library from the network, without needing to set up anything aside from a webserver. Jinzora can also be configured as a jukebox such that (if the hosting computer has speakers) you can remotely control playback of any of your music.

The setup was fairly straightforward. There were a few technical issues configuring PHP to have enough memory and enough execution time to run; it certainly isn’t a plug and play experience. Specifically, for Jinzora to run well, it requests longer than 300 seconds on max execution time and 32+ megs for a memory limit.

Those two things gave me worries, because typically upping the execution time and memory limit are the lazy programmer’s way out. I actually did bump into a memory limit during the import of my music library and had to set it to 128M for things to go through.

Aside from that, the web-install was fairly easy. There weren’t any technical issues that stopped the show (aside from needing to bump up the memory limit). When the importer encountered oddly tagged or oddly named files (with special characters or whatever), it did throw errors, but it kept going. It took approximately an hour to import 15,000 files.

Once it was running, I was pleased with some neat things it provided: there are related artists links for any particular track/artist. These are broken apart into artists you have and ones you don’t. Searching seemed to work reasonably well. Streaming worked without a hitch — clicking a music link prompted me asking if I wanted to play what the site was sending (it sends .m3u files, so they should be playable by most modern music players.)

It finds album art if there is art in the music directory in question. I think there’s some mechanism for grabbing album art from external sources, but couldn’t find the setting.

You can configure Jinzora to send useage reports to last.fm if you use their services.

Unfortunately, I didn’t fully explore this application due to the one showstopper.

The showstopper, and reason why I won’t be using it, is this:it is insanely slow.

Granted, the machine I ran this on was a PII 300 w/ 256M RAM, but even so, it felt as if the application were iterating over the file structure each time I loaded a page. Page loads literally took up to 30 seconds to load. For comparison, a simple rails application I wrote, working from a MySQL database over the same library, took half a second to load (with the crappy Rails paginator, no less.) As well, a PHP-based indexer I wrote that actually traverses the directory on each page load takes only 10 seconds to load. I realize that the hosting machine is crap, but it’s not that crap. Something is definitely wrong with the code.

(This is where it gets a little more technical.)

So… I looked at the database Jinzora set up to see if I could optimize with indices or something. There are indices. The fields they are indexing are varchars. In fact, every single field is a varchar in any of the tables, with the exception of playcounts.

The primary key is a varchar. I repeat: the primary key is a varchar!

In despair, I looked at the PHP to see how rough it would be to fork Jinzora for proper design and normalization.

Every select that used the primary key (which was called ‘path’ and used the file’s path, btw) used the LIKE statement. There was nothing that looked like a JOIN statement (and come to think of it, I don’t really know if you can join on text fields in all the databases it supports.)

Every single SELECT looked something like “SELECT foo WHERE path LIKE ‘$path%’ “. It’s not just using text fields to index, it’s using LIKE, which is perhaps the slowest mechanism for accessing data in a database!

Searching on the net, I found some forum user asking how to speed up Jinzora. He reported that he was seeing 150 queries per second. FOR A SINGLE PAGELOAD.

No wonder my experience was so slow. (Aside: Actually, it’s kind of supportive of the Linux system’s configuration and PHP’s performance that it didn’t take 5 minutes to load with that kind of crap going on behind the scenes.)

Anyway, that was my experience with Jinzora. I saw a Google cached post that hinted at database normalization in version 3, so maybe I’ll return to it at a later date. For now I’ll just write my own.

• Don’t use the client linked from your Citrix login page, it’s probably version 7 but the latest is version 10. Go here, instead
• You will need Open Motif. Use synaptic to search for ‘motif’. Install limbotif3, libmotif-dev, and motifnls
• You may need to set up your FontPath in your xorg.conf. This is the “Files” section of /etc/X11/xorg.conf
  
FontPath "/usr/share/X11/fonts/cyrillic"
FontPath "/usr/share/X11/fonts/100dpi"
FontPath "/usr/share/X11/fonts/75dpi"
FontPath "/var/lib/defoma/x-ttcidfont-conf.d/dirs/TrueType"
FontPath "/usr/share/fonts/X11/100dpi/:unscaled"
FontPath "/usr/share/fonts/X11/75dpi/:unscaled"
FontPath "/usr/share/fonts/X11/100dpi/:unscaled"
FontPath "/usr/share/fonts/X11/Type1"
FontPath "/usr/share/fonts/X11/misc"

• You also need the correct security certificates, because apparently they don’t all ship with the client (I kept getting errors about UTN-USERFirst-Hardware not being trusted). I copied all my mozilla certs and the cacerts.org and that seemed to do the trick:
  
sudo cp /usr/share/ca-certificates/cacert.org/cacert.org.crt /usr/lib/ICAClient/keystore/cacerts/
sudo cp /usr/share/ca-certificates/mozilla/* /usr/lib/ICAClient/keystore/cacerts/


That worked for me. Good luck!

Situation: class with dynamic, on the fly class method definitions of the form

What this does is defines some class methods role1_email_addresses, role2_email_addresses, etc on the fly.

Works perfectly, until the first element returned in users is actually an empty email.

This is an example of writing clever code for succinctness (which is rampant in ruby circles). The problem is this code has bugs, and in order to fix it you have to rewrite it or add extra checks and workarounds. Which means the code is no longer succinct nor clever.

In my recent experience, that’s all that overly-clever code gets you. You just end up making extra work for yourself (or the maintainer) down the line.

• The settings migration was excellent. The installer transferred all my settings like a charm. The only exception was the custom login screen and splash image. Not such a big deal, but worth noting.
• Also awesome, it saw that I had one of the Broadcom-based wireless cards and automatically ran the firmware cutter to install the Windows driver for the Linux kernel. That was sort of handy, I must say.
• Wireless configuration was way easier with fewer lags.
• Compiz worked out of the box with no messing with settings.

Now, the bad:

• Even though wireless was easily configured, it refused to work. I was never able to get the broadcom card to work with ndis nor the firmware cutter in the first place — at least not on networks requiring a WEP key. Furthermore, the USB wireless card I use instead requires a Realtek driver that is automatically blacklisted, due to hanging the kernel, apparently. So, no network for me. Wired works, but… meh, it’s a laptop you know?
• Beryl appeared to work, but there was some tweaking necessary. Supposedly there’s an xorg.conf trick one can do to make the non-refreshing windows (that I was experiencing) go away, but I never tried it.
• Even with a wired connection (which worked) I could not open an SSH network place in Nautilus to anywhere. It almost acted as if it were disabled.

So, I blew all that away and reinstalled Edgy. I know that works and I know I can get network/beryl running correctly under that. Really all I was interested in with 7.04 was the networking stuff and virtualization support. I didn’t try the latter, but the former kind of put a kabosh on the whole thing.

One thing that’s interesting to note: now that I have 6.10 reinstalled I’m noting some neat things. Everything is a lot zippier. Wireless works at my workplace for the first time. I consistenly had problems with mysql working, and now I don’t. It’s pretty awesome.

Anyway, I guess the message I’m taking away is that the Feisty beta really is beta software. I suppose I’ve had my view of “beta software” modified from companies like Google consistently releasing fully-functional beta code. This OS beta harkens back to the days when “beta” really meant something.

So take all that for what you will. My advice, though: wait until the April for the full release.


foreach ($row as $k => $v) {
 if (is_numeric($v))
  continue;
$row[$k] = preg_replace("/([\\xC0-\\xDF])([\w|\W])/e", "'\\1'.chr(ord('\\2')-64) ", $v) ;
}

Incidentally, had to do som strange copy/paste action to get that line to not break wordpress.

And FWIW, that preg_replace effectively converts windows-1252 characters out of the unused overlap range in ISO 8859-1.

Sometimes you just want some Whitesnake or Ratt to rock out to. “Big, dumb music” as one of my buddies calls it. If you’re spending all your music time only listening to stuff that means something deep and profound, I kind of think that you’re taking it way too seriously.

It’s supposed to be fun and entertaining, yo.

• Use the internal editor to edit your .htaccess. If you do it outside WP, appparently you lose your changes the next time you do ANY editing or posting. Seems like a bug that it resaves the .htaccess each time you post or edit a post. Dunno.
• 
  This ends up being problematic, because if you mess up the .htaccess, you could get a 500 internal server error, preventing you from fixing the problem.
• Toggle your .htaccess from Anyone Can Write to No One Can Write, if you need custom changes. WordPress by design overwrites this file whenever you do anything. This means when you change an option, post something, or edit something. It’s really stupid.
• Make sure you set your blog address/wordpress address in the Options->General section.
• Make sure your hosting provider or your .htaccess restrict to accessing this url. For example, if you have your address at http://www.foo.com/, make sure people cannot access or are redirected when they go to http://foo.com/, and vice versa. This solves a lot of problems with ajax stuff.
• The rich editor is really nice, but if you have a lot of content and a lot of nested divs, sometimes it will incorrectly parse them and try to close what’s already been closed. Use that feature with caution if you handcode your html, frequently.

For the .htaccess stuff, the following may be pasted to provide redirection. Use option 1 or 2 depending on your needs. This will only work if your host has mod_rewrite. You may need to put “RewriteEngine On” before the RewriteCond lines.
(Pretty much stolen from here):

#Option 1 Redirects domain.com to www.domain.com
RewriteCond %{HTTP_HOST} !^www\.domain\.name [NC]
RewriteCond %{HTTP_HOST} !^$
RewriteRule ^/(.*) http://www.domain.name/$1 [L,R]

# Option 2 Redirects www.domain.com to domain.com
RewriteCond %{HTTP_HOST} ^www\.domain\.name [NC]
RewriteCond %{HTTP_HOST} !^$
RewriteRule ^/(.*) http://domain.name/$1 [L,R]


Note that for wp2, you should make sure you have a way of deleting/modifying your .htaccess file in case you screw something up. If you try to edit the .htaccess outside of wordpress, wordpress2 won’t know about the changes and will overwrite them so you need to use the Manage->Files section to edit. However, if you break your rewrite because you mis-pasted, you’ll need a way to revert that doesn’t involve WP2.

Example:
SELECT t.* FROM table t WHERE t.id IN (SELECT DISTINCT f.id FROM table f, table e WHERE f.something = e.somethingelse AND f.someotherthing = e.somethingdifferent)


In Postgres and MySQL, IN seems to be fairly efficient and treats the subselect as a simple list. In SQL Server, it appears that IN does some kind of massive join, or perhaps a series of SELECTs times the number of rows in the superselect’s table.

All I know is that code works fine with little latency under pgsql and mysql, but at a certain row limit SQL Server is an order of magnitude different — we’re talking 10 seconds for a select versus half a second. The solution is to not use IN — do a JOIN, instead.

Just thought I’d pass that on to all you DB peeps.

(And for the record, I like IN statements because they seem more semantic and readable to me.)

This took me a little bit to figure out, cause I was a little slow, last night. Also, the Google Map API documentation doesn’t seem to cover this sort of eventuality.

Say that, for whatever reason, you have an array of GMarker objects and you want to programatically apply an info window to each of them.

Doing so is not as straightforward as this code:

for (i in markers) {
...
GEvent.addListener(markers[i], "mouseover", function() {
marker[i].openInfoWindowHtml(text_var);
} );
}


This won’t work, because in the third parameter, “i” is passed in literally. In order to have the “i” successfully expanded to an index, you have to eval the function creation, like so:

for (i in markers) {
...
GEvent.addListener(marker[i], "mouseover", eval('function() {' +
'marker[' + i + '].openInfoWindowHtml("' + text_var + '"); ' +
'} ')
);
}


Just passing it on.

If you have to write multi-line comments documenting the return value or behavior of a language function, that function is poorlyl named or poorly implemented.

By way of example, I direct you to xml_parse_into_struct

When that breakpoint fires, it’s like you’re hanging off the edge of the universe.

On a related note, I have the sneaking suspicion that this is bad news, but it solved a particularly vexing problem with validations firing on belongs_to models even when those models weren’t created yet in the build process (e.g. the leaf model needed to validate first, creation of the parent was dependent on the leaf. If the leaf didn’t validate, the parent necessarily won’t (and shouldn’t attempt to) validate because it hasn’t built the relation yet):

In the leaf

validates_numericality_of :some_value,
:allow_nil => false,
:if => Proc.new { !self.reflect_on_association(:transactions).nil? }

The scary thing for me is that I’m not sure why it necessarily works.

If you need to automate closing comments in WordPress and aren’t afraid to get your hands a little dirty with some editing, read on.

It’s a basic hack to the comments file. Go to wp_content/themes/<themename>/comments.php

somewhere there should be two lines that look sort of like
<? if ('open' == $post->comment_status) : ?>

Modify those lines to something like
<? if (('open' == $post-> comment_status) && strtotime($post->post_date ." + 30 days") >= strtotime(date('F d, Y H:i:s'))) : ?>

Next, you need to change the post verification:
Go to wp_content/wp-comments-post.php. At the very top should be a line that includes
"SELECT post_status, comment_status..."
click between post_status and comment_status, add
post_date,
(include the comma)

Then a few lines down change
} elseif ( 'closed' == $status->comment_status ) {

to


} elseif (
('closed' == $status->comment_status) ||
(strtotime($status->post_date ." + 30 days") < strtotime(date('F d, Y H:i:s'))
)

Note that you can change the "30 days" to whatever interval you like. What this does is check if the post date is within 30 days of the current date as well as checking the status of comment posting. If it's not within 30 days, the test fails and the comments box isn't displayed.

I'm posting this not cause it's a neat hack, but because it's kind of daunting to find where crap is located in WordPress, sometimes.

Also, this kind of thing looks like it'll work under WP2 (the file looks the same to me). I don't have a working install to test it, though.

(p.s. Keep in mind that if you change your theme, you'll have to redo these changes in whatever theme folder you've selected.)