Introduction to Theory of Computing Lecture Notes (Cornell CS4810)

46 Pages·2017·1.15 MB·English

by  it-ebooks

CS 4810 — Fall'13
Syllabus| Lectures| Homework| Resources
Lecture 1: Boolean Functions
Boolean values
There are two Boolean values, TRUE and FALSE. For convenience, we will
1 0
denote these values as and . Boolean values are often also called bits. We
will use Boolean values to encode information.
Basic Boolean operations
Basic Boolean operations are functions that take one or two Boolean values as
input and output a Boolean value. We will focus the following operations:
1  if x = 1,
(x) := {
NOT
0  otherwise.
1  if x = 1 and y = 1,
(x,y) := {
AND
0  otherwise.
1  if x = 1 or y = 1,
(x,y) := {
OR
0  otherwise.
The following notation is common:
¬x for  (x),
NOT
x∧y for  (x,y),
AND
x∨y for  (x,y).
OR
Truth table
We can represent the Boolean operations NOT, AND, OR by the following truth
table.
x y (x) (x,y) (x,y)
NOT AND OR
0 0 1 0 0
0 1 1 0 1
1 0 0 0 1
1 1 0 1 1
De Morgan’s Laws
There are redundancies among the operations NOT, AND, OR. In particular, we
can write AND in terms of NOT, OR. Similarly, OR can be expressed in terms of
NOT, AND.
¬(x∧y) = ¬x∨¬y,
¬(x∨y) = ¬x∧¬y.
We can check that these identities hold by looking at the above truth table.
Boolean functions
n f n
An -bit Boolean function is a function that take Boolean values as
input and outputs a Boolean value, i.e.,
f:{0,1}n → {0,1}.
Boolean function can express interesting algorithmic questions:
1  if x is (the binary encoding of) a prime number,
(x) := {
PRIMESn
0  otherwise.
(x)
RIEMANNn
:=
1  if x is (the binary encoding of) a proof of the Riemann hypothesis,
{
0  otherwise.
(The Riemann hypothesis is an important open question in mathematics. If you
resolve the hypothesis, you get a Millennium Prize.)
We will try to understand whether these kind of functions can be expressed
by a small number of basic Boolean operations. (It turns out that both functions
above can be expressed by a number of basic Boolean operations that is
n
polynomial in .)

CS 4810 — Fall'13
Syllabus| Lectures| Homework| Resources
Lecture 2: Boolean Circuits
Boolean circuit
n 1
A Boolean circuit with inputs and output is a directed acyclic graph that
consists of the following kind of nodes:
1,…,n
» Input nodes, labeled , without ingoing edges (sources).
» Any number of gates, labeled with NOT, AND, OR.
» All AND, OR gates have two incoming edges.
» All NOT gates have one incoming edge.
» One output gate, without outgoing edges (sink). (All other gates have at least
one outgoing edge.)
Circuit computation
x ,…,x
Suppose we assign Boolean values 1 n to the input nodes of a circuit.
Then, we can propagate these values through the gates of the circuit by
applying the corresponding Boolean operations.
More formally, we would process the gates of the circuit in topological
u
order. To process a gate labeled with a Boolean operation
T ∈ { , , } T
NOT AND OR , we apply the operation to the values assigned to the
u u
ancestors of and we assign the output of this operation to . (Since we
u
process nodes in topological order, the ancestors of have already been
u
processed at the time we process .)
C n 1 n
We say a Boolean circuit with inputs and output computes an -bit
f:{0,1}n → {0,1}
Boolean function if for every input
x = (x ,…,x ) ∈ {0,1}n f(x)
1 n , the function value is equal to the value
C x ,…,x
propagated to the output gate of when we assign 1 n to the input
C
nodes of .
One can show (e.g., by induction) that a circuit computes only one function.
In particular, the function does not depend on the order we process the gates.
f C
We write C to denote the function computed by a circuit .
Example: parity
n
Consider the following -bit Boolean function
1  if x +…+x  is an odd number,
(x) := { 1 n
PARITYn
0  otherwise.
n = 2 , ,
For , we can express PARITY in terms of NOT AND OR as follows
(x ,x ) = (x ∧¬x )∨(¬x ∧x )
PARITY2 1 2 1 2 1 2
2
The parity of bits is called exclusive-or. A common notation for
(x ,x ) x ⊕x
PARITY2 1 2 is 1 2.
n 2
If is a power of , we can construct a circuit for PARITY recursively based
on the following identity
(x) = (x ,…,x )⊕ (x ,…,x ).
PARITYn PARITYn/2 1 n/2 PARITYn/2 n/2+1 n
Circuit complexity
For a given Boolean functions, there could be many ways of computing it by a
circuit. But some circuits are better than others.
C
We measure the efficiency of a circuit by its size
size(C) := the number of gates of C.
f
For a Boolean function , we define its size as the minimum size of circuit
f
that computes .
Another measure for the efficiency of a circuit is its depth
depth(C)
:= length of longest path from an input node to the output gate

Generic upper bound
n f
Theorem: For every -bit Boolean function there exists a circuit of
10⋅2n −10
size at most to compute it.
n f
Proof: We will use induction on . If we fix one of input bit of the function ,
n−1
then we are left with an -bit Boolean function.
n = 1 1
For , the theorem is true because for every -bit function, we can find
10 n > 1 f f n−1
a circuit of size at most . Suppose . Let 0 and 1 be the two bit
f
functions obtained by fixing the first bit of , i.e.,
f (x ,...,x ) = f(0,x ,...,x ),
0 2 n 2 n
f (x ,...,x ) = f(1,x ,...,x ).
1 2 n 2 n
f f n−1
Both 0 and 1 are bit functions. The following identity allows us to
f f f
express the original function in terms of 0 and 1 as well as a few basic
operations
f = (¬x ∧f )∨(x ∧f ).
1 0 1 1
f f
Therefore, we can construct a circuit for from a circuit for 0 and a
f 4
circuit for 1 as well as additional gates (one NOT gate, one OR gate, and two
size(f) ≤ size(f )+size(f )+4
AND gates). It follows that 0 1 . By induction
size(f ) size(f ) 10⋅2n−1 −10
hypothesis, both 0 and 1 are at most . By
combining these bound, we get
size(f) ≤ 10⋅2n −20+4 = 10⋅2n −16 ≤ 10⋅2n −10.
CS 4810 — Fall'13
Syllabus| Lectures| Homework| Resources
Lecture 3: Hard functions
Generic lower bound
n ∈ N n ≥ 100 n
Theorem: For every with , there exists an -bit Boolean
2n/100n
function that requires circuits of size at least .
n
Proof: The idea is to compare the number of -bit Boolean functions
f:{0,1}n → {0,1} s = 2n/100n
to the number of circuits of size at most . If
the number of functions is larger than the number of such circuits, it means
that there exists a functions that is not computed by such a circuit.
n
Let us first count the number of -bit Boolean functions. In general, the
X Y |Y ||X|
number of functions from a finite set to a finite set is equal to . (You
n
should convince yourself of this fact.) Hence, the number of -bit Boolean
22n
functions is .
s s
Now let’s count the number of size- circuits. A circuit of size consists of a
s′ = s+n ≤ 2s n s
directed acyclic graph (DAG) on nodes ( input nodes and
2
gates). Each node has in-degree at most . (The in-degree of a node is the
0
number of ingoing edges.) Furthermore, each input node (in-degree ) is
i ∈ [n]
labeled with an input variable . Each gate is labeled with a Boolean
operation, either AND, OR, or NOT.
We claim that we can describe such a circuit completely by a bit string of
10slogs
length at most . First, we give every node a unique ID. Since there are
s′ logs′
only nodes in the circuit, we only need bits for the ID. (We can use the
1 s′ s′
numbers from to as IDs.) Our description of the circuit will consist of
10logs′
parts, each of length at most (one part per vertex). To describe a single
u u logs′
vertex , we store the ID of ( bits), the IDs of the vertices that have
u 2logs′ u
incoming arcs to ( bits) and the label of , which is either a Boolean
logn ≤ logs′
function or the index of an input/output variable (at most bits).
We don’t need to store the outgoing arcs of a vertex, because those will be
4logs′
stored as incoming arcs by other vertices. In total, we store at most bits
per vertex. Hence, the description of the whole circuit is indeed shorter than
10slogs
bits.
10 log
s 210slogs
It follows that the number of circuits of size is at most (the
10slogs s = 2n/100n
number of bit strings of length ). For , this number is
22n
much less than . (Check this calculation.) It follows that an overwhelming
n 2n/100n
majority of -bit Boolean functions does not have circuits of size .
\
Remark: While the theorem shows that some functions require large
circuits, the proof of the theorem does not give an explicit construction of such
a function. (The proof is non-constructive.) It’s a famous open problem to show
strong lower bounds on the circuit size of concrete functions. For example, the
following funtion is conjectured to require circuits of exponential size
⎧0  if x is the binary encoding of a circuit
(x) := ⎨  that computes the constant 0 function,
CIRCUIT−SATn ⎩
1  otherwise.
Straight-line Programs
In this part, we will discuss an exact correspondence between circuits and a
simple kind of programs.
Here is an example of a straight-line program:
x ,x
Input: 1 2
Instructions:
y := ¬x
» 1 1
y := ¬x
» 2 2
y := x ∧y
» 3 1 2
y := y ∧x
» 4 1 2
y := y ∨y
» 5 3 4
y
Output: 5
This program outputs the parity of the two input bits. We can construct a circuit
that corresponds to this program in the following way: For every instruction,
we have a gate in the circuit that applies the same Boolean operation and has
incoming edges from nodes corresponding to the right-hand-side variables of
the instruction. The size of the resulting circuit is exactly equal to the number
of instructions of the straight-line program.
Definition: A straight-line program is a sequence of instructions of the
following form: Each instruction applies a basic Boolean operation (
, ,
NOT AND OR) to input variables or previously assigned variables and
assigns the outcome of this operation to a new variable. The output of
the program is the outcome of last instruction. The length of a straight-
line program is the number of instructions.
ℓ
Exercise: A Boolean function has a straight-line program of length at most if
ℓ
and only if it has a Boolean circuit of size at most .
CS 4810 — Fall'13
Syllabus| Lectures| Homework| Resources
Lecture 4: Deterministic Finite Automata
The following figure shows the state diagram of a deterministic finite
automaton (DFA),
0 1
0
1
q₀ q₁ q₂
0,1
q q q q
The automaton has three states 0, 1, and 2. The state 0 is designated as the
q
start state and 1 is designated as an accept state. The directed edges between
states specify the transitions of the automaton when reading an input string.
1
This automaton accepts the set of strings that contain and have an even
0 1
number of ’s after the last .
Languages
Σ Σ = {0,1} Σ = a,b,c
An alphabet is a finite set of symbols, e.g., or . (We
w Σ
will usually content ourselves with the binary alphabet.) A string over is a
w = x ⋯x x ,…,x ∈ Σ
finite sequence of symboles, written as 1 n for 1 n . The
ε
empty string (string of length zero) is denoted . The set of all strings is denoted
Σ∗ L L ⊆ Σ∗
. A language is any subset of strings, so that .
L
We will use languages to encode computational problems. (A language
f:Σ∗ → {0,1} f(w) = 1
can be thought to encode the function with if and
w ∈ L
only if ).

A,B ⊆ Σ∗
Operations on languages: Let be two languages. The following
operations on languages are useful:
A∪B A∩B
» Basic set-theoretic operations, union , intersection , and
¯A¯¯¯ := Σ∗ ∖A
complementation . (These operations correspond to
, ,
OR AND NOT.)
AB := {uv ∣ u ∈ A,v ∈ B}
» Concatentation .
Ai i
» Kleene star $A^*:=A0A1A^2 $, where denotes the -fold concatenation of
A i = 0,1
with itself. (The corner cases are defined so that the rule
AiAj = Ai+j A0 = {ε} A1 = A
holds. Concretely, and ).