Top-K Queries limit a result set to a handful of records that have the highest weighting factors. This is useful in decision support and in web services queries where you want to have a controlled selection of data returned, and you do not want to maintain query state, such as a cursor, in the service.

If you are in a setting where "weight" is an indexed value, this can be a reasonably efficient query. If the intermediary set is small, then this query may also be efficient. However, if either is not true, then the query can be very inefficient.

SELECT TOP

In some SQL environments, the expression SELECT TOP k is implemented. SQL Server, for example.

            SELECT TOP k url, weight FROM data ORDER BY weight DESC

For other database systems, the canonical Top-K query looks like:

SELECT url, weight FROM

(SELECT url, weight FROM data ORDER BY weight DESC)

WHERE rownum < k;

Inside Knowledge - Mapping Example

In case of mapping, weight may be a distance calculation such as with a mapping application finding coffee shops near an address. That distance "weight" would certainly not be keyed because it would vary with each query.

With added inside information, you can enhance the query. Using the map example, if you have a table of grid coordinates and the states that touch within that grid coordinate, you can first query which states touch grid coordinates within your search circle, and then using the indexed value of state to reduce the query, perform weight calculations on that reduced set.

That grid sub-query would work for mapping purposes works because you have inside information on the data, knowing what is queried, and how the weighting may be optimized with native field information.

The grid was used as an example. Other geo-location information can be given by zip codes or latitude and longitude references compared to the coverage of individual states.

Interruptible, Without Inside Knowledge

What if you do not have inside knowledge of the data being queried? This approach may be useful to help conserve resources and produce a query with acceptable results.

One thing to avoid is a large intermediary sub-query result in a high-throughput web services engine. If many users are simultaneously creating sorted multi-million record results to get a Top-100 list, then large amounts of virtual memory are being used inefficiently.

In an iterative measure, perform the following (pseudo Java-like procedural SQL):

First, get a base set of records:

TopK = SELECT url, weight FROM data WHERE rownum <= k;

TopKMax = SELECT MAXIMUM weight FROM TopK;

Then, iterate while the collection is being refined:

            while(true) {

            TopKTemp = SELECT url, weight FROM data

WHERE rownum <= k AND weight > TopKMax; // (A)

            TopK = TopK + TopKMax ;     // combine the records

            if(count(SELECT * FROM TopKTemp) = 0) {

                        return TopK;    // (B)

                        }

            if(timeForQueryExpired)           // (C) allow for time limits

                        return TopK;

TopK = SELECT * FROM

(SELECT UNIQUE url, weight

 FROM TopK ORDER BY weight DESC)

WHERE rownum <= k;

            }

Notes:

• For services considerations, time limits (C) can be imposed to assure that a result will be produced within a timeout period.

• Because ORDER BY clause is not specified in the large query, the intermediary resources required to process the query for sorting are not needed. Database systems tend to optimize their queries to look in cached data first before retrieving more records from the disk. Multiple requests querying on the Top-K may grow their intermediary results together even if the queries between the clients are not identical.

• If a timeout occurs, then the highest ranking entries may not be returned because it may not have been discovered yet.

• Using the refining value "weight > TopKMax", (A) duplicate values of high-scoring entries may not be gathered. At point (B), a back-fill approach to find intermediary values may be desired.

1.      Find k - 1 values where weight = TopKMax

2.      If fewer than (k - 1) are found, then look between the second highest value (Top2) and TopKMax, "WHERE weight >= Top2 and weight < TopKMax)", and add those in.

3.      Continue downward until all K in the set are the largest.

Individual UTF-8 Character and Extensions

The standard UTF-8 format describes how characters are encoded. These are noted in the rows that display the form "UTF-8".

Extending the UTF-8 encoding to encodings that are hinted at by the format are represented by "UTF-8-EX", UTF-8 extensions. These show five- and six-byte representations. These are by default ways to encode integers up to 10 decimal digits long.

UTF-8-OB encodings show how the out-of-band encodings can occur. These formats allow for non-character encodings of arbitrarily long form. The minimum length of these forms is two bytes.

UTF-END, coded by 11111111 (FF) is special in the above encoding. At no time, will the byte oxff be used for anything other than UTF-END, and cannot occur within any other encoding.

The complete list of encodings above is called UTF-8-XX. All encodings that except the UTF-END code together form the set of the UTF-8-FF characters, or x-FF for short.

Overall Document Format

VERSION

Version is the first field in the payload.

The version field contains one UTF-8 Extended Character.

PROMOTE

• UTF-8 characters define a numeric code for characters which may be represented by 1-, 2-, 3-, or 4-byte encryptions. If a character is not used frequently, with this scheme, the UTF-8 representation for that character can be replaced with a numeric representation. For example:

00000010 is a one-byte UTF-8 encoding.

11000000 10000010 is the same character in two-byte encodings, thereby freeing the value 00000010 to be used for another, more common value.

• When a character is promoted, it allows the character value to be used for another form. If it is promoted but not allocated in the lexical section, it is an integer. Otherwise, it is a tag, or a lexical value.

METATAG

Metatags provide the structure for a document. Since each character represents uniquely a metatag, lexical constant, or integer, the metatags can be identified to reintroduce structure. If the <section> metatag is met before the remaining metatags, then that means the constructs aren't used. The metatags used are:

FORMAT

• Much information in XML is related to numerical values. Format provides a way to separate the format of a number from a numeric value which can be easily represented. For example:

123-45-6789 and 234-45-3456 are both SSNs. They have a format "9‑99‑9999" and two values: 123456789 and 234453456. A nine-digit number can be represented by with 5 byte UTF-8-OB number. The format "9‑99‑9999" may be entered into a lexical dictionary with a short code if it is commonly used. The overall usage is more compressed.

• Formats have three format values. These are "9", "a", and "A". "a" and "A" define whether a hexadecimal value should be reported in upper or lower case. The leftmost format value identifies that it should be repeated for the remainder of the integer. "$999" would be the same as "$9". A format of "9" is the implied format when reproducing a number"
• Negative numbers are represented through the formatting process "-9" is the standard representation for a negative number. "(9)" may be an alternate representation.
• Time may represented as "9:99am", and "9:99pm" and 5:52pm would be:

<fmt> <"9:99pm"> <552>.

• Since the format identifies whether the value is a hexadecimal value or a decimal value, for hexadecimal values, they are calculated (mod 5), a == 0, f == 4. Thus, hexadecimal values are numerically smaller though there may be more formats. Having the format and the integer in hand, the representational value will be deterministic.

Examples:

<fmt> <"a9"> <"11"> à "b01"

<fmt><"aa"><"11"> à <"cb">

Simulations of both forms showed this to be more compressed. As a result, there may be more formats for hexadecimal numbers. If all combinations of a six-digit hexadecimal number are used, then there will be 64 formats

SPACES

In the concatenation of values, the differentiation between values concatenated directly and those that are separated by an intermediate space character are recognized. The overall metatag can identify if the spaces should be included as part of the tag, and when the x-terms are concatenated, the whitespace will be inserted between them automatically. The code supports the <sws> metatag to switch that mode.

When a format is used within a concatenation expression, the format is expanded, and then put into the concatenation.

LEXICAL

Within an XML document, the same phrase may be repeated many times. For example, attribute names, tag names, namespace tags, "blue", and so on. Numbers that are used many times may be placed here and given a more concise numeric code.

BODY

The body is an implicit concatenation field, without automatic spaces, and continues for the remainder of the document. For an XML document, it may begin with something like:

<?xml version="1.0" encoding="ISO-8859-1"?>

For this, the XML format will be understood so tag "?xml" will have an end tag "?". Encoding will appear as follows:

<tag> <"?xml"> <atr> <"version"> <fmt> <"9.9"> <10> <atr> <"encoding"> <fmt> <"ISO-9-9"> <88591> <FF>

Duality of Numbers

When overridden, a character can represent a number that has two forms. For example, UTF-8 character 01100001 10000000 can represent 128, as can UTF-8-OB encoding 10000001 00000000.  When a range is specified in the PROMOTE section of the document, either character codes or numbers can be promoted. The range specified after the promotion will always be in the UTF-8-OB format, and the range will always stay within the sequence of UTF-8 characters or UTF-8-OB encodings.

UTF-8-OB numbers cannot be overridden, however, just with UTF-8 characters, they may be remapped in the lexical overlay section.

Process of Encoding

The following is a simple encoding:

Part 1: traverse the document and make a lexical encoding for each unique tag, each unique attribute name, and each namespace reference. For document transparency, these values cannot be distilled further. No tag or attribute name can be represented in a concatenated form. This is a requirement of the representation to ease introspection.

Part 2: factor all attribute values, child data, and comments with numeric formats. The rules:

Numbers must have two digits within a span of not more than twice as many characters. For this purpose, "a" through "f" and "A" through "F" are non-alphabetic.

Inventory the different formats and the value that is applied to the format.

Part 3: construct a distribution of the factored values and the distribution of the factors. The highest occurring tags, integers, characters, formats, etc. will have the shortest representations possible.

Part 4: For the first 128 most common values, select whether it should be promoted. Are there 1-byte characters that should not be characters? Often, 0 through 10 hex are not used, so they will be promoted automatically. Select those that should be promoted and map the most common values to them. If there's a natural integer form, e.g. 100 is in the collection and character 100 is promoted, then assign the natural mapping. If the natural mapping is used, then there will not need to be a mapping for "100" in the lexical mapping section of the document.

Part 5: Repeat for the two-byte and three-byte UTF-8 characters. For standard US and European documents, all of these may be promoted (since they are not used), and the character codes can be reused.

Note: Reserve enough space in the two-byte formats to hold all the meta-tag definitions. Later, when a firm estimate of metatag requirements is known, then it will require less juggling to promote 1-byte characters or select a 3-byte or 4-byte representation for the metatags.

Part 6: The mappings of metatags, tags, attributes, comments, and values is reasonably complete. Look at all the lexical mappings which have an occurrence of one, and are not tags. These are not to be put into the lexical mappings. Instead, the characters will be put out in the multi- forms of the value expressions.

Note: We're looking at the weighted average of declarations versus instantiations. "10" 300 times will require 600 bytes in natural literal form (as characters), 600 bytes in the UTF-8-OB form, and 303 bytes if 10 is mapped to character 10.

The number 65600 200 times will require 1000 bytes in natural form, 600 bytes in UTF-8-OB form, and 304 bytes if 65600 is mapped to a one-byte character such as 10. (The size of the representational form) * (the frequency) + (encoding overhead) = representation cost.

Part 7: readjust the values for the metatags to accurately reflect how commonly they are used.

Simulations have shown that metatags, if used, will nearly always be one-byte, two-bytes, or three since the document size to span more than but 20 three-byte characters and three-byte UTF-8-OB integers would be so large that the metatags, if used, would either be more frequent than the average token, or not used at all. To that end, any metatag that is used more frequently than the least frequently-used of the 128 most-used codes will be swapped for that code.

Part 8: Output the form.

Process of Decoding

Part 1: Read the first code. This will reveal the version code.

Reading the first code is done by reading the first byte, if it starts with "10", then continue reading bytes in the UTF-8-OB format until, and including, the byte beginning with "0" is found. If the a character begins with something other than "10", read it as if it were a UTF-8 or UTF-8-EX character using the leading number of 1s to identify how many bytes long it is.

Part 2: Read in the second code. This will reveal the section code to separate the sections.

Part 3: Read codes until the next section code is found. With each code read, assign to the next in the sequence of metatags. If the section code is found before the list of metatags has exhausted, then those metatags are not used in the encoding.

Part 4: Read pairs of UTF-8-FF, and UTF-8-OB to determine which encodings are replaced by lexical representations or numeric overcodings.