# SqlTokenizer

> SqlTokenizer splits an SQL query into individual SQL tokens.

[API reference](/docs/api/index.md) › [yii](/docs/api/yii/index.md) › [db](/docs/api/yii/db/index.md) › SqlTokenizer

| abstract class | `yii\db\SqlTokenizer` |
| --- | --- |
| Extends | [`yii\base\Component`](/docs/api/yii/base/component/index.md) » [`yii\base\BaseObject`](/docs/api/yii/base/baseobject/index.md) |
| Implements | [`yii\base\Configurable`](/docs/api/yii/base/configurable/index.md) |
| Since | 2.0.13 |
| Source | [`framework/db/SqlTokenizer.php`](https://github.com/yiisoft/yii2/blob/22.0/framework/db/SqlTokenizer.php#L32) |

1 subclass:

- [`yii\db\sqlite\SqlTokenizer`](/docs/api/yii/db/sqlite/sqltokenizer/index.md)

SqlTokenizer splits an SQL query into individual SQL tokens.

It can be used to obtain an addition information from an SQL code.

Usage example:

```
$tokenizer = new SqlTokenizer("SELECT * FROM user WHERE id = 1");
$root = $tokeinzer->tokenize();
$sqlTokens = $root->getChildren();
```

Tokens are instances of [`SqlToken`](/docs/api/yii/db/sqltoken/index.md).

## Properties

| Property | Type | Description | Defined by |
| --- | --- | --- | --- |
| <span id="property-behaviors"></span>[`$behaviors`](/docs/api/yii/base/component/index.md#property-behaviors) read-only | `array<`[`yii\base\Behavior`](/docs/api/yii/base/behavior/index.md)`<static>>` | List of behaviors attached to this component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="property-sql"></span>`$sql` | `string` | SQL code. Default: `null`. | `SqlTokenizer` |
| <span id="property-length"></span>`$length` protected | `int` | SQL code string length. Default: `null`. | `SqlTokenizer` |
| <span id="property-offset"></span>`$offset` protected | `int` | SQL code string current offset. Default: `null`. | `SqlTokenizer` |

## Methods

| Method | Description | Defined by |
| --- | --- | --- |
| [`__construct()`](#method-__construct) | Constructor. | `SqlTokenizer` |
| <span id="method-__call"></span>[`__call()`](/docs/api/yii/base/component/index.md#method-__call) | Calls the named method which is not a class method. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-__clone"></span>[`__clone()`](/docs/api/yii/base/component/index.md#method-__clone) | This method is called after the object is created by cloning an existing one. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-__get"></span>[`__get()`](/docs/api/yii/base/component/index.md#method-__get) | Returns the value of a component property. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-__isset"></span>[`__isset()`](/docs/api/yii/base/component/index.md#method-__isset) | Checks if a property is set, i.e. defined and not null. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-__set"></span>[`__set()`](/docs/api/yii/base/component/index.md#method-__set) | Sets the value of a component property. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-__unset"></span>[`__unset()`](/docs/api/yii/base/component/index.md#method-__unset) | Sets a component property to be null. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-attachBehavior"></span>[`attachBehavior()`](/docs/api/yii/base/component/index.md#method-attachBehavior) | Attaches a behavior to this component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-attachBehaviors"></span>[`attachBehaviors()`](/docs/api/yii/base/component/index.md#method-attachBehaviors) | Attaches a list of behaviors to the component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-behaviors"></span>[`behaviors()`](/docs/api/yii/base/component/index.md#method-behaviors) | Returns a list of behaviors that this component should behave as. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-canGetProperty"></span>[`canGetProperty()`](/docs/api/yii/base/component/index.md#method-canGetProperty) | Returns a value indicating whether a property can be read. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-canSetProperty"></span>[`canSetProperty()`](/docs/api/yii/base/component/index.md#method-canSetProperty) | Returns a value indicating whether a property can be set. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-className"></span>[`className()`](/docs/api/yii/base/baseobject/index.md#method-className) static | Returns the fully qualified name of this class. | [`yii\base\BaseObject`](/docs/api/yii/base/baseobject/index.md) |
| <span id="method-detachBehavior"></span>[`detachBehavior()`](/docs/api/yii/base/component/index.md#method-detachBehavior) | Detaches a behavior from the component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-detachBehaviors"></span>[`detachBehaviors()`](/docs/api/yii/base/component/index.md#method-detachBehaviors) | Detaches all behaviors from the component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-ensureBehaviors"></span>[`ensureBehaviors()`](/docs/api/yii/base/component/index.md#method-ensureBehaviors) | Makes sure that the behaviors declared in [`behaviors()`](/docs/api/yii/base/component/index.md#method-behaviors) are attached to this component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-getBehavior"></span>[`getBehavior()`](/docs/api/yii/base/component/index.md#method-getBehavior) | Returns the named behavior object. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-getBehaviors"></span>[`getBehaviors()`](/docs/api/yii/base/component/index.md#method-getBehaviors) | Returns all behaviors attached to this component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-hasEventHandlers"></span>[`hasEventHandlers()`](/docs/api/yii/base/component/index.md#method-hasEventHandlers) | Returns a value indicating whether there is any handler attached to the named event. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-hasMethod"></span>[`hasMethod()`](/docs/api/yii/base/component/index.md#method-hasMethod) | Returns a value indicating whether a method is defined. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-hasProperty"></span>[`hasProperty()`](/docs/api/yii/base/component/index.md#method-hasProperty) | Returns a value indicating whether a property is defined for this component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-init"></span>[`init()`](/docs/api/yii/base/baseobject/index.md#method-init) | Initializes the object. | [`yii\base\BaseObject`](/docs/api/yii/base/baseobject/index.md) |
| <span id="method-off"></span>[`off()`](/docs/api/yii/base/component/index.md#method-off) | Detaches an existing event handler from this component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-on"></span>[`on()`](/docs/api/yii/base/component/index.md#method-on) | Attaches an event handler to an event. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| [`tokenize()`](#method-tokenize) | Tokenizes and returns a code type token. | `SqlTokenizer` |
| <span id="method-trigger"></span>[`trigger()`](/docs/api/yii/base/component/index.md#method-trigger) | Triggers an event. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| [`indexAfter()`](#method-indexAfter) protected | Returns an index after the given string in the SQL code starting with the specified offset. | `SqlTokenizer` |
| [`isComment()`](#method-isComment) protected abstract | Returns whether there's a commentary at the current offset. | `SqlTokenizer` |
| [`isIdentifier()`](#method-isIdentifier) protected abstract | Returns whether there's an identifier at the current offset. | `SqlTokenizer` |
| [`isKeyword()`](#method-isKeyword) protected abstract | Returns whether the given string is a keyword. | `SqlTokenizer` |
| [`isOperator()`](#method-isOperator) protected abstract | Returns whether there's an operator at the current offset. | `SqlTokenizer` |
| [`isStringLiteral()`](#method-isStringLiteral) protected abstract | Returns whether there's a string literal at the current offset. | `SqlTokenizer` |
| [`isWhitespace()`](#method-isWhitespace) protected abstract | Returns whether there's a whitespace at the current offset. | `SqlTokenizer` |
| [`startsWithAnyLongest()`](#method-startsWithAnyLongest) protected | Returns whether the longest common prefix equals to the SQL code of the same length at the current offset. | `SqlTokenizer` |
| [`substring()`](#method-substring) protected | Returns a string of the given length starting with the specified offset. | `SqlTokenizer` |

## Method details

### __construct() <span id="method-__construct"></span>

```php
public function __construct(string $sql, array $config = [])
```

Constructor.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$sql` | `string` | SQL code to be tokenized. |
| `$config` | `array` | Name-value pairs that will be used to initialize the object properties |

### tokenize() <span id="method-tokenize"></span>

```php
public function tokenize(): yii\db\SqlToken
```

Tokenizes and returns a code type token.

**Returns**

[`SqlToken`](/docs/api/yii/db/sqltoken/index.md): Code type token.

### indexAfter() <span id="method-indexAfter"></span>

```php
protected function indexAfter(string $string, int|null $offset = null): int
```

Returns an index after the given string in the SQL code starting with the specified offset.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$string` | `string` | String to be found. |
| `$offset` | `int\|null` | SQL code offset, defaults to current if `null` is passed. |

**Returns**

`int`: Index after the given string or end of string index.

### isComment() <span id="method-isComment"></span>

```php
abstract protected function isComment(int &$length): bool
```

Returns whether there's a commentary at the current offset.

If this methos returns `true`, it has to set the `$length` parameter to the length of the matched string.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$length` | `int` | Length of the matched string. |

**Returns**

`bool`: Whether there's a commentary at the current offset.

### isIdentifier() <span id="method-isIdentifier"></span>

```php
abstract protected function isIdentifier(int &$length, string &$content): bool
```

Returns whether there's an identifier at the current offset.

If this methos returns `true`, it has to set the `$length` parameter to the length of the matched string.
It may also set `$content` to a string that will be used as a token content.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$length` | `int` | Length of the matched string. |
| `$content` | `string` | Optional content instead of the matched string. |

**Returns**

`bool`: Whether there's an identifier at the current offset.

### isKeyword() <span id="method-isKeyword"></span>

```php
abstract protected function isKeyword(string $string, string &$content): bool
```

Returns whether the given string is a keyword.

The method may set `$content` to a string that will be used as a token content.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$string` | `string` | String to be matched. |
| `$content` | `string` | Optional content instead of the matched string. |

**Returns**

`bool`: Whether the given string is a keyword.

### isOperator() <span id="method-isOperator"></span>

```php
abstract protected function isOperator(int &$length, string &$content): bool
```

Returns whether there's an operator at the current offset.

If this methos returns `true`, it has to set the `$length` parameter to the length of the matched string.
It may also set `$content` to a string that will be used as a token content.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$length` | `int` | Length of the matched string. |
| `$content` | `string` | Optional content instead of the matched string. |

**Returns**

`bool`: Whether there's an operator at the current offset.

### isStringLiteral() <span id="method-isStringLiteral"></span>

```php
abstract protected function isStringLiteral(int &$length, string &$content): bool
```

Returns whether there's a string literal at the current offset.

If this methos returns `true`, it has to set the `$length` parameter to the length of the matched string.
It may also set `$content` to a string that will be used as a token content.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$length` | `int` | Length of the matched string. |
| `$content` | `string` | Optional content instead of the matched string. |

**Returns**

`bool`: Whether there's a string literal at the current offset.

### isWhitespace() <span id="method-isWhitespace"></span>

```php
abstract protected function isWhitespace(int &$length): bool
```

Returns whether there's a whitespace at the current offset.

If this methos returns `true`, it has to set the `$length` parameter to the length of the matched string.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$length` | `int` | Length of the matched string. |

**Returns**

`bool`: Whether there's a whitespace at the current offset.

### startsWithAnyLongest() <span id="method-startsWithAnyLongest"></span>

```php
protected function startsWithAnyLongest(array<int, string>|array<int, array<string, mixed>> &$with, bool $caseSensitive, int|null &$length = null, string|null &$content = null): bool
```

Returns whether the longest common prefix equals to the SQL code of the same length at the current offset.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$with` | `array<int, string>\|array<int, array<string, mixed>>` | Strings to be tested. The method **will** modify this parameter to speed up lookups. |
| `$caseSensitive` | `bool` | Whether to perform a case sensitive comparison. |
| `$length` | `int\|null` | Length of the matched string. |
| `$content` | `string\|null` | Matched string. |

**Returns**

`bool`: Whether a match is found.

### substring() <span id="method-substring"></span>

```php
protected function substring(int $length, bool $caseSensitive = true, int|null $offset = null): string
```

Returns a string of the given length starting with the specified offset.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$length` | `int` | String length to be returned. |
| `$caseSensitive` | `bool` | If it's `false`, the string will be uppercased. |
| `$offset` | `int\|null` | SQL code offset, defaults to current if `null` is passed. |

**Returns**

`string`: Result string, it may be empty if there's nothing to return.