# SqlTokenizer

> SqlTokenizer splits SQLite query into individual SQL tokens.

[API reference](/docs/api/index.md) › [yii](/docs/api/yii/index.md) › [db](/docs/api/yii/db/index.md) › [sqlite](/docs/api/yii/db/sqlite/index.md) › SqlTokenizer

| class | `yii\db\sqlite\SqlTokenizer` |
| --- | --- |
| Extends | [`yii\db\SqlTokenizer`](/docs/api/yii/db/sqltokenizer/index.md) » [`yii\base\Component`](/docs/api/yii/base/component/index.md) » [`yii\base\BaseObject`](/docs/api/yii/base/baseobject/index.md) |
| Implements | [`yii\base\Configurable`](/docs/api/yii/base/configurable/index.md) |
| Since | 2.0.13 |
| Source | [`framework/db/sqlite/SqlTokenizer.php`](https://github.com/yiisoft/yii2/blob/22.0/framework/db/sqlite/SqlTokenizer.php#L20) |

SqlTokenizer splits SQLite query into individual SQL tokens.

It's used to obtain a `CHECK` constraint information from a `CREATE TABLE` SQL code.

See also: <https://www.sqlite.org/draft/tokenreq.html>, <https://sqlite.org/lang.html>.

## Properties

| Property | Type | Description | Defined by |
| --- | --- | --- | --- |
| <span id="property-behaviors"></span>[`$behaviors`](/docs/api/yii/base/component/index.md#property-behaviors) read-only | `array<`[`yii\base\Behavior`](/docs/api/yii/base/behavior/index.md)`<static>>` | List of behaviors attached to this component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="property-sql"></span>[`$sql`](/docs/api/yii/db/sqltokenizer/index.md#property-sql) | `string` | SQL code. Default: `null`. | [`yii\db\SqlTokenizer`](/docs/api/yii/db/sqltokenizer/index.md) |
| <span id="property-length"></span>[`$length`](/docs/api/yii/db/sqltokenizer/index.md#property-length) protected | `int` | SQL code string length. Default: `null`. | [`yii\db\SqlTokenizer`](/docs/api/yii/db/sqltokenizer/index.md) |
| <span id="property-offset"></span>[`$offset`](/docs/api/yii/db/sqltokenizer/index.md#property-offset) protected | `int` | SQL code string current offset. Default: `null`. | [`yii\db\SqlTokenizer`](/docs/api/yii/db/sqltokenizer/index.md) |

## Methods

| Method | Description | Defined by |
| --- | --- | --- |
| <span id="method-__construct"></span>[`__construct()`](/docs/api/yii/db/sqltokenizer/index.md#method-__construct) | Constructor. | [`yii\db\SqlTokenizer`](/docs/api/yii/db/sqltokenizer/index.md) |
| <span id="method-__call"></span>[`__call()`](/docs/api/yii/base/component/index.md#method-__call) | Calls the named method which is not a class method. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-__clone"></span>[`__clone()`](/docs/api/yii/base/component/index.md#method-__clone) | This method is called after the object is created by cloning an existing one. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-__get"></span>[`__get()`](/docs/api/yii/base/component/index.md#method-__get) | Returns the value of a component property. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-__isset"></span>[`__isset()`](/docs/api/yii/base/component/index.md#method-__isset) | Checks if a property is set, i.e. defined and not null. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-__set"></span>[`__set()`](/docs/api/yii/base/component/index.md#method-__set) | Sets the value of a component property. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-__unset"></span>[`__unset()`](/docs/api/yii/base/component/index.md#method-__unset) | Sets a component property to be null. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-attachBehavior"></span>[`attachBehavior()`](/docs/api/yii/base/component/index.md#method-attachBehavior) | Attaches a behavior to this component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-attachBehaviors"></span>[`attachBehaviors()`](/docs/api/yii/base/component/index.md#method-attachBehaviors) | Attaches a list of behaviors to the component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-behaviors"></span>[`behaviors()`](/docs/api/yii/base/component/index.md#method-behaviors) | Returns a list of behaviors that this component should behave as. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-canGetProperty"></span>[`canGetProperty()`](/docs/api/yii/base/component/index.md#method-canGetProperty) | Returns a value indicating whether a property can be read. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-canSetProperty"></span>[`canSetProperty()`](/docs/api/yii/base/component/index.md#method-canSetProperty) | Returns a value indicating whether a property can be set. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-className"></span>[`className()`](/docs/api/yii/base/baseobject/index.md#method-className) static | Returns the fully qualified name of this class. | [`yii\base\BaseObject`](/docs/api/yii/base/baseobject/index.md) |
| <span id="method-detachBehavior"></span>[`detachBehavior()`](/docs/api/yii/base/component/index.md#method-detachBehavior) | Detaches a behavior from the component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-detachBehaviors"></span>[`detachBehaviors()`](/docs/api/yii/base/component/index.md#method-detachBehaviors) | Detaches all behaviors from the component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-ensureBehaviors"></span>[`ensureBehaviors()`](/docs/api/yii/base/component/index.md#method-ensureBehaviors) | Makes sure that the behaviors declared in [`behaviors()`](/docs/api/yii/base/component/index.md#method-behaviors) are attached to this component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-getBehavior"></span>[`getBehavior()`](/docs/api/yii/base/component/index.md#method-getBehavior) | Returns the named behavior object. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-getBehaviors"></span>[`getBehaviors()`](/docs/api/yii/base/component/index.md#method-getBehaviors) | Returns all behaviors attached to this component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-hasEventHandlers"></span>[`hasEventHandlers()`](/docs/api/yii/base/component/index.md#method-hasEventHandlers) | Returns a value indicating whether there is any handler attached to the named event. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-hasMethod"></span>[`hasMethod()`](/docs/api/yii/base/component/index.md#method-hasMethod) | Returns a value indicating whether a method is defined. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-hasProperty"></span>[`hasProperty()`](/docs/api/yii/base/component/index.md#method-hasProperty) | Returns a value indicating whether a property is defined for this component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-init"></span>[`init()`](/docs/api/yii/base/baseobject/index.md#method-init) | Initializes the object. | [`yii\base\BaseObject`](/docs/api/yii/base/baseobject/index.md) |
| <span id="method-off"></span>[`off()`](/docs/api/yii/base/component/index.md#method-off) | Detaches an existing event handler from this component. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-on"></span>[`on()`](/docs/api/yii/base/component/index.md#method-on) | Attaches an event handler to an event. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-tokenize"></span>[`tokenize()`](/docs/api/yii/db/sqltokenizer/index.md#method-tokenize) | Tokenizes and returns a code type token. | [`yii\db\SqlTokenizer`](/docs/api/yii/db/sqltokenizer/index.md) |
| <span id="method-trigger"></span>[`trigger()`](/docs/api/yii/base/component/index.md#method-trigger) | Triggers an event. | [`yii\base\Component`](/docs/api/yii/base/component/index.md) |
| <span id="method-indexAfter"></span>[`indexAfter()`](/docs/api/yii/db/sqltokenizer/index.md#method-indexAfter) protected | Returns an index after the given string in the SQL code starting with the specified offset. | [`yii\db\SqlTokenizer`](/docs/api/yii/db/sqltokenizer/index.md) |
| [`isComment()`](#method-isComment) protected | Returns whether there's a commentary at the current offset. | `SqlTokenizer` |
| [`isIdentifier()`](#method-isIdentifier) protected | Returns whether there's an identifier at the current offset. | `SqlTokenizer` |
| [`isKeyword()`](#method-isKeyword) protected | Returns whether the given string is a keyword. | `SqlTokenizer` |
| [`isOperator()`](#method-isOperator) protected | Returns whether there's an operator at the current offset. | `SqlTokenizer` |
| [`isStringLiteral()`](#method-isStringLiteral) protected | Returns whether there's a string literal at the current offset. | `SqlTokenizer` |
| [`isWhitespace()`](#method-isWhitespace) protected | Returns whether there's a whitespace at the current offset. | `SqlTokenizer` |
| <span id="method-startsWithAnyLongest"></span>[`startsWithAnyLongest()`](/docs/api/yii/db/sqltokenizer/index.md#method-startsWithAnyLongest) protected | Returns whether the longest common prefix equals to the SQL code of the same length at the current offset. | [`yii\db\SqlTokenizer`](/docs/api/yii/db/sqltokenizer/index.md) |
| <span id="method-substring"></span>[`substring()`](/docs/api/yii/db/sqltokenizer/index.md#method-substring) protected | Returns a string of the given length starting with the specified offset. | [`yii\db\SqlTokenizer`](/docs/api/yii/db/sqltokenizer/index.md) |

## Method details

### isComment() <span id="method-isComment"></span>

```php
protected function isComment(mixed &$length): bool
```

Returns whether there's a commentary at the current offset.

If this methos returns `true`, it has to set the `$length` parameter to the length of the matched string.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$length` | `mixed` | Length of the matched string. |

**Returns**

`bool`: Whether there's a commentary at the current offset.

### isIdentifier() <span id="method-isIdentifier"></span>

```php
protected function isIdentifier(mixed &$length, mixed &$content): bool
```

Returns whether there's an identifier at the current offset.

If this methos returns `true`, it has to set the `$length` parameter to the length of the matched string.
It may also set `$content` to a string that will be used as a token content.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$length` | `mixed` | Length of the matched string. |
| `$content` | `mixed` | Optional content instead of the matched string. |

**Returns**

`bool`: Whether there's an identifier at the current offset.

### isKeyword() <span id="method-isKeyword"></span>

```php
protected function isKeyword(mixed $string, mixed &$content): bool
```

Returns whether the given string is a keyword.

The method may set `$content` to a string that will be used as a token content.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$string` | `mixed` | String to be matched. |
| `$content` | `mixed` | Optional content instead of the matched string. |

**Returns**

`bool`: Whether the given string is a keyword.

### isOperator() <span id="method-isOperator"></span>

```php
protected function isOperator(mixed &$length, mixed &$content): bool
```

Returns whether there's an operator at the current offset.

If this methos returns `true`, it has to set the `$length` parameter to the length of the matched string.
It may also set `$content` to a string that will be used as a token content.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$length` | `mixed` | Length of the matched string. |
| `$content` | `mixed` | Optional content instead of the matched string. |

**Returns**

`bool`: Whether there's an operator at the current offset.

### isStringLiteral() <span id="method-isStringLiteral"></span>

```php
protected function isStringLiteral(mixed &$length, mixed &$content): bool
```

Returns whether there's a string literal at the current offset.

If this methos returns `true`, it has to set the `$length` parameter to the length of the matched string.
It may also set `$content` to a string that will be used as a token content.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$length` | `mixed` | Length of the matched string. |
| `$content` | `mixed` | Optional content instead of the matched string. |

**Returns**

`bool`: Whether there's a string literal at the current offset.

### isWhitespace() <span id="method-isWhitespace"></span>

```php
protected function isWhitespace(mixed &$length): bool
```

Returns whether there's a whitespace at the current offset.

If this methos returns `true`, it has to set the `$length` parameter to the length of the matched string.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `$length` | `mixed` | Length of the matched string. |

**Returns**

`bool`: Whether there's a whitespace at the current offset.