What are Tokens in programming

Smallest individual element of a program is called as Token. Everything you see inside a program is a token. There are generally five types of tokens:

  1. Keyword
  2. Identifier
  3. Operator
  4. Separator
  5. Literal

What are Keywords?

Keyword is a reserved word whose meaning is already defined by the programming language. We cannot use keyword for any other purpose inside programming. Every programming language have some set of keywords.
Examples: int, do, while, void, return etc(Note: These keywords are common to C and C influenced languages).

What are Identifiers?

Identifiers are the name given to different programming elements. Either name given to a variable or a function or any other programming element, all follow some basic naming conventions listed below:

  1. Keywords must not be used as an identifier.
  2. Identifier must begin with an alphabet a-z A-Z or an underscore_ symbol.
  3. Identifier can contains alphabets a-z A-Z, digits 0-9 and underscore _ symbol.
  4. Identifier must not contain any special character (e.g. !@$*.'[] etc.) except underscore _.

Examples of some valid identifiers

num, Num, _num, _Num, num1, Num1, _num1, _Num1, _1num, _1Num, _num_, number_to_add

etc.

Examples of some invalid identifiers

1num, number to add, 1_num, num-to-add, num@

etc.

What are Operator?

Operators are the symbol given to any arithmetical or logical operations. Various programming languages provides various sets of operators some common operators are:
Lets suppose two variables a=10, b=5
- operator subtracts second operand from first i.e. a - b and results 5.

Operator Description Example
Arithmetic operator
Arithmetic operator are used to perform basic arithmetic operations.
+ Adds two operand. a + b gives 15
* Multiplies two operands. a * b gives 50
/ Divides two operands. a / b gives 2
% Modulus operator divides the first operand from second and returns the remainder. It is generally used for checking divisibility. a % b gives 0 (As 10/5 will have 0 remainder)
Assignment operator
Assignment operator is used to assign value to a variable. The value is assigned from right to left.
= Assigns value from right operand to left operand. a = 10 will assign 10 in a

Relational operator

Relational operator are used to check relation between any two operands. Whether any of them is greater, equal or not equal.

> If value of left operand is greater than right, returns true else returns false (a > b) will return true
< If value of right operand is greater than left, returns true else returns false (a < b) will return false
== If both operands are equal returns true else false (a == b) will return false
!= If both operands are not equal returns true else false. (a != b) will return true
>= If value of left operand is greater or equal to right operand, returns true else false (a >= b) will return true
<= If value of right operand is greater or equal to left operand, returns true else false (a <= b) will return false
Logical operator
Logical operator are used to combine two boolean expression together and results in a single boolean value according to the operand and operator used.
&& Used to combine two expressions. If both operands are true or Non-Zero, returns true else false ((a>=1) && (a<=10)) will return true since (a>=1) is true and also (a<=10) is true.
|| If any of the operand is true or Non-zero, returns true else false ((a>1) || (a<5)) will return true. As (a>1) is true. Since first operand is true hence there is no need to check for second operand.
! Logical NOT operator is a unary operator. Returns the complement of the boolean value. !(a>1) will return false. Since (a>1) is true hence its complement is false.
Bitwise operator
Bitwise operator performs operations on Bits(Binary level). Lets suppose a = 10, b = 5
a = 0000 1010 (8-bit binary representation of 10)
b = 0000 0101 (8-bit binary representation of 5)
& Bitwise AND performs anding operation on two binary bits value. If both the values are 1 then will result is 1 else will result in 0.
  0000 1010
& 0000 0101
____________
  0000 0000
| Bitwise OR returns 1 if any of the two binary bits are 1 else returns 0.
  0000 1010
| 0000 0101
___________
  0000 1111
^ Bitwise XOR returns 1 if both the binary bits are different else returns 0.
  0000 1010
^ 0000 0101
___________
  0000 1111
~ Bitwise COMPLEMENT is a unary operator.It returns the complement of the binary value i.e. if the binary bit is 0 returns 1 else returns 0.
~ 0000 1010
___________
  1111 0101
<< Bitwise LEFT SHIFT operator is also unary operator. It shift the binary bits to the left. It inserts a 0 bit value to the extreme right of the binary value. Or we may say it generally multiplies the value with 2.
  0000 1010 << 2 
= 0010 1000
>> Bitwise RIGHT SHIFT operator is an unary operator. It shifts the binary bits to the right. It inserts a 0 bit value to the extreme left of the binary value. Or we may say it generally divides the value with 2.
  0000 1010 << 2
= 0000 0010
Increment/Decrement operator
Increment/Decrement operator is a unary operator used to increase an integer value by 1 or decrease it by 1. Increment/decrement operator are of two types Postfix and Prefix.
++ Increment operator will add 1 to an integer value. a++ will give 11
++a will also give 11
-- Decrement operator will subtract 1 from an integer value. a-- will give 9
--a will also give 9
Conditional/Ternary operator
Ternary operator as a conditional operator and is similar to simple if-else. It takes three operand.
?: It is used as conditional operator. Syntax of using ternary operator:
(condition) ? (true part) : (false part)
b = (a>1) ? a : b;
will store the value 10 in b as (a>1) is true hence true part will execute, assigning the value of a in b.

What are Separators?

Separators are used to separate different programming elements. The various types of separators used in programming are:
 (Space) \t(Tab) \n(New line) . , ; () {} []

What are Literals?

Literals are constant values that are used for performing various operations and calculations. There are basically three types of literals:

  1. Integer literal

    An integer literal represents integer or numeric values.
    Example: 1, 100, -12312 etc

  2. Floating point literal

    Floating point literal represents fractional values.
    Example: 2.123, 1.02, -2.33, 13e54, -23.3 etc

  3. Character literal

    Character literal represent character values. Single character are enclosed in a single quote(' ') while sequence of character are enclosed in double quotes(" ")
    Example: 'a', 'n', "Hello", "Hello123" etc.

Write your doubts or suggestion. I will try my best to help. You must escape source code before commenting. To format your source code paste your source code inside
<pre><code> ----Your Source Code---- </code></pre>