This comes to be a problem in many scenarios, specially when we are ordering by a position field that can contain an integer value for its position on the dataset or null if position is not defined. Rows that have undefined position have lower weight than the specified ones thus coming first.

The following SQL query is from a very common scenario that represents a SELECT to fetch all city registries in “importance” (most common, not in fact important cities – don’t get mad if you live in an odd city) order.

SELECT
  id, city
FROM
  cities
ORDER BY
  POSITION;

This brings us all null-valued position rows first and not null positioned in ascending order, at the bottom.

This happens because our ordering pool will look like the following:
position (integer or null), city field value (string)

So null values are considered smaller than 1 (lowest positive integer) and then comes first in our resultset.

Workaround

Workaround is never a good choice, but I’m listing here so you DON’T, EVER do that.

Using a huge integer value instead of null ones

Setting the default field value to a big integer so it will come always on the bottom of the list. This is particularly bad where it compromises your database integrity since your data will have a fake value instead of a null value. null means “not set”, “undefined”. Setting it otherwise will not mean the same. Database integrity is important so your data can be handled and transported by other apps. You may not bother with this now, but you may in the future.

Doing it maintaining data structure and integrity

We will take advantage of CASE native from most DBMS.

SELECT
  id, city
FROM
  cities
ORDER BY
  CASE WHEN POSITION IS NULL THEN 1 ELSE 0 END,
  POSITION,
  city

NOTE: Field city is added on ordering list in behalf of sorting in ascending order the rows with undefined position

CASE is a good SQL way to implement flow-control under database-level. In this case, if position field is null CASE returns 1, 0 otherwise.

This will make the order pool to be like:
(0/1), position (integer or null), city field value (string)

This way, other ordering criteria position and city will be treated as secondary and tertiary respectively, depending on the (0,1) value. 1 is bigger than 0 so if all null values are treated as 1, they will come after the ones that has a position integer representation that will come with 0 as first criteria.

Applying this concept on big tables

As CASE must do a calculation for every single row we may (and probably will) encounter speed issues when handling this ordering on big tables.

A solution is to mantain a boolean field like has_position and use it on the first criteria so our order pool gets something like:
has_position, position (integer or null), city field value (string)

The SQL code as it follows:

SELECT
  id, city
FROM
  cities
ORDER BY
  has_position,
  POSITION,
  city

NOTE: See which comes first (true/false) on boolean ordering on your DBMS and apply the ASC/DESC clause to has_position accordingly

The drawback is that you will have to run once in a while (or on position change) a SQL query to update the registries:

UPDATE cities SET has_position = 1 WHERE POSITION IS NOT NULL;
UPDATE cities SET has_position = 0 WHERE POSITION IS NULL;

You can even use CASE for this, but will have the same speed issues.

UPDATE cities SET has_position = CASE WHEN POSITION IS NULL THEN 0 ELSE 1 END;

The difference is that you will have this issue just time to time instead of every query.

Benchmarking

We tested the three methods: only with position field (that doesn’t returns data in the way we want), with has_position, position (best form proposed for big tables) and with the CASE trick.

The results are as expected:

Exactly:

position

25,77 secs

has_position, position

12,83 secs

CASE trick

23,20 secs

The tests were run on a HP Pavillion DV6780SE with a Core2 Duo 1.66ghz, 3GB ram with 2.11.3deb1ubuntu1.1 mysql running under Ubuntu Linux 8.04 on a table (without indexes) with 249.751 rows.

Download the test files

Test files include

positioning.sql

Database schema

wordlist_pt_br.txt

pt_BR Wordlist (use you preferred wordlist here)

mysql_case_speed_test.php

Structured programmed benchmark Script. Call with ?populate to populate database and generate random position numbers.

Download order-with-null-values.zip

Whats your nifty trick?

No related posts.

So we have data and when it gets bigger and bigger, we need to search through it. Let’s take some approaches to the Search function and find out the better way.

The Search Scope: Who will be using it and what they look for

The answer is short: people (do you hear the sound of evil?)
People has the natural-born fear of technology so we can expect that the search terms will be as less-accurate as possible. Imagine your user sitting before your search box and thinking “What should I put on this little box to get what I want?” and then putting a single word about it.

Example:
The user wants to find about a pretty neat Mont-Blanc pen.
The possible keywords this user will enter are: mont-blanc and pen (unless the user is high, there isn’t much more keywords to serve)

Let’s assume he doesn’t wan’t to be that specific (may be looking for other models) and search only for pen.

The Search Methods: The Bomb and the Rifle

The Bomb: The LIKE method
Widely diffused, the LIKE method is the most popular since it can be ran in any table within any datatype. This method uses a comparison function present in many DBMS that searches for anything matching a * (everything) pattern on the side specified with a percentage (%) symbol.

For example:

SELECT *
FROM keyword
WHERE keyword LIKE '%pen%'; -- will get *pen*

NOTE: PostgreSQL has an implementation called ILIKE that performs the same on a case-insensitive manner.

Drawbacks:
Speed issues
This method runs on a data retrieving method called SEQ SCAN (or Sequential Scan). It basically runs through all the table comparing your query with the data, one by one. It takes time. In this tiny case (13 records) it took 0.0009s (0.9ms). Big tables’ nightmare.

Relevancy issues
As it only returns rows that matches in any way, you can’t get a keyword density analysis before data has being retrieved and thus show less-relevant products to the client. Business owners’ nightmare.

Senseness issues
Imagine our pen example. Could you imagine what results pen* could bring up? I think our Mont-blanc user will need some parental control. Users’ nightmare.

That is why I call him ‘The Bomb’ since it returns you loads of data with short accuracy.

The Rifle Method: Indexing words
This method is what the Google Search Era represents now. Relevance, best-matching, keyword density.
When we say ‘We’ll have to wait until Google indexes…’ or ‘My page is poorly indexed…’ that means this indexing: Breaking your text into words and saving its position (starting from 0) on the field that will be searched for and storing it on a keyword index table. The basic keyword index table formula is the following: id-of-content-registry,the-keyword,the-field-where-keyword-is-located and the-position-of-the-keyword-on-the-field's-content. So when we search for ’007′ we perform a exact-match query:

SELECT *
FROM keyword_index
WHERE keyword = 'pen'; -- only exact pen match

NOTE: Surely Google analyzes much more than the position on the pages’ content such as context, semanticsness etc.

Gains:
Speed issues
This query (in the same tiny case (13 records)) took 0.0006s (0.6ms) what is an insignificant difference on this scenario with few records that will show up broadly when you reach high volume of data. Big tables’ heaven

Relevancy Issues
With the position field being stored we have info of each time the keyword appeared on a text. Performing a simple COUNT() will feed us with the information of how many times that keyword appears on the text (density) so we can sort via relevancy to the end-user. Business owners’ heaven

Senseness Issues
In this method we may get Mont-blanc pen, Lamy pen (and no Penthouse and other non-user-friendly things). Users’ heaven

The indexed systems may vary. You may have the position field or not. You may have even more fields. Depends on your search criteria.

So, depends the size of the target you wanna reach.
The Rifle (Indexed) search method gives you more speed and accuracy, The Bomb (LIKE) search method gives your more results.

Choose your weapon!

No related posts.

Googled for the answer and found two good pieces of code:

If you want it simple, like ucfirst(), getting only the very first letter of the first word, you may have:

UPDATE `table` SET target_field =
        CONCAT(
                UCASE(SUBSTRING(`source_field`, 1, 1)),
                LOWER(SUBSTRING(`source_field`, 2))
        );

Just replace ‘table’ with the table’s name and ‘target_field’ with your… target field?

If you want the job done in all terms, Joe Zack’s MySQL Capitalize Function may fit you as well as it fit me.

Who’s got a better idea?

No related posts.